onadata.libs.utils package¶
Submodules¶
onadata.libs.utils.api_export_tools module¶
API export utility functions.
- onadata.libs.utils.api_export_tools.custom_response_handler(request, xform, query, export_type, token=None, meta=None, dataview=False, filename=None, metadata=None)¶
Returns a HTTP response with export file for download.
- onadata.libs.utils.api_export_tools.export_async_export_response(request, export)¶
Checks the export status and generates the reponse :param request: :param export: :return: response dict example {“job_status”: “Success”, “export_url”: …}
- onadata.libs.utils.api_export_tools.external_export_response(export)¶
Redirects to export_url of XLSReports successful export. In case of a failure, returns a 400 HTTP JSON response with the error message.
- onadata.libs.utils.api_export_tools.generate_google_web_flow(request)¶
Returns a OAuth2WebServerFlow object from the request redirect_uri.
- onadata.libs.utils.api_export_tools.get_async_response(job_uuid, request, xform, count=0)¶
Returns the status of an async task for the given job_uuid.
- onadata.libs.utils.api_export_tools.get_existing_file_format(data, format)¶
Util function to extract the existing form extension
- onadata.libs.utils.api_export_tools.get_metadata_format(data_value)¶
Returns metadata format/extension
- onadata.libs.utils.api_export_tools.include_hxl_row(dv_columns, hxl_columns)¶
This function returns a boolean value. If the dataview’s columns are not part of the hxl columns, we return False. Returning False would mean that we don’t have to add the hxl column row if there aren’t any hxl columns in the dataview. :param dv_columns - dataview columns :param hxl_columns - hxl columns from the dataview’s xform
:return True or False
- onadata.libs.utils.api_export_tools.log_export(request, xform, export_type)¶
Logs audit logs of export requests.
- onadata.libs.utils.api_export_tools.process_async_export(request, xform, export_type, options=None)¶
Check if should generate export or just return the latest export. Rules for regenerating an export are:
Filter included on the exports.
New submission done.
- Always regenerate external exports.
- (External exports uses templates and the template might have
changed)
- Parameters:
request –
xform –
export_type –
options – additional export params that may include query: export filter token: template url for xls external reports meta: metadataid that contains the external xls report template url remove_group_name: Flag to determine if group names should appear
- Returns:
response dictionary
- onadata.libs.utils.api_export_tools.response_for_format(data, format=None)¶
Return appropriately formatted data in Response().
onadata.libs.utils.async_status module¶
- onadata.libs.utils.async_status.async_status(status, error=None)¶
- onadata.libs.utils.async_status.celery_state_to_status(state)¶
onadata.libs.utils.audit module¶
onadata.libs.utils.backup_tools module¶
Backup utilities.
- onadata.libs.utils.backup_tools.create_zip_backup(zip_output_file, user, xform=None)¶
Create a ZIP file with a user’s XForms and submissions.
- onadata.libs.utils.backup_tools.restore_backup_from_path(dir_path, username)¶
Only restores xml submissions, media files are assumed to still be in storage and will be retrieved by the filename stored within the submission
- onadata.libs.utils.backup_tools.restore_backup_from_xml_file(xml_instance_path, username)¶
Creates submission instances in the DB from a submission XML file.
- onadata.libs.utils.backup_tools.restore_backup_from_zip(zip_file_path, username)¶
Restores XForms and submission instances from a ZIP backup file.
onadata.libs.utils.briefcase_client module¶
ODK BriefcaseClient utils module
- class onadata.libs.utils.briefcase_client.BriefcaseClient(url, username, password, user)¶
Bases:
object
ODK BriefcaseClient class
- download_instances(form_id, cursor=0, num_entries=100)¶
Download the XML submissions.
- download_manifest(manifest_url, id_string)¶
Downloads the XForm manifest from an ODK server.
- download_media_files(xml_doc, media_path)¶
Downloads media files from an ODK server.
- download_xforms(include_instances=False)¶
Downloads the XForm XML form an ODK server.
- push()¶
Publishes XForms and XForm submissions.
- onadata.libs.utils.briefcase_client.django_file(file_obj, field_name, content_type)¶
Return an InMemoryUploadedFile file object.
- onadata.libs.utils.briefcase_client.node_value(node, tag_name)¶
Returns the first nodeValue of of an elementin the node with the matching tag_name otherwise returns empty list [].
onadata.libs.utils.cache_tools module¶
Cache utilities.
- onadata.libs.utils.cache_tools.reset_project_cache(project, request, project_serializer_class)¶
Clears and sets project cache
- onadata.libs.utils.cache_tools.safe_delete(key)¶
Safely deletes a given key from the cache.
- onadata.libs.utils.cache_tools.safe_key(key)¶
Return a hashed key.
onadata.libs.utils.chart_tools module¶
Chart utility functions.
- onadata.libs.utils.chart_tools.build_chart_data(xform, language_index=0, page=0)¶
Returns chart data for all the fields in the
xform
.
- onadata.libs.utils.chart_tools.build_chart_data_for_field(xform, field, language_index=0, choices=None, group_by=None, data_view=None)¶
Returns the chart data for a given field.
- onadata.libs.utils.chart_tools.build_chart_data_from_widget(widget, language_index=0)¶
Returns chart data from a widget.
- onadata.libs.utils.chart_tools.calculate_ranges(page, items_per_page, total_items)¶
Return the offset and end indices for a slice.
- onadata.libs.utils.chart_tools.find_choice_label(choices, string)¶
Returns the choice label of the given
string
.
- onadata.libs.utils.chart_tools.get_chart_data_for_field(field_name, xform, accepted_format, group_by, field_xpath=None, data_view=None)¶
Get chart data for a given xlsform field.
- onadata.libs.utils.chart_tools.get_choice_label(choices, string)¶
string is the name value found in the choices sheet.
Select one names should not contain spaces but some do and this conflicts with Select Multiple fields which use spaces to distinguish multiple choices.
A temporal fix to this is to search the choices list for both the full-string and the split keys.
- onadata.libs.utils.chart_tools.get_field_choices(field, xform)¶
Retrieve field choices from a form survey element :param field: :param xform: :return: Form field choices
- onadata.libs.utils.chart_tools.get_field_from_field_name(field_name, xform)¶
Returns the field if the
field_name
is in thexform
.
- onadata.libs.utils.chart_tools.get_field_from_field_xpath(field_xpath, xform)¶
Returns the field if the
field_xpath
is in thexform
.
- onadata.libs.utils.chart_tools.get_field_label(field, language_index=0)¶
Returns the
field
’s label or name based on selectedlanguage_index
.’
- onadata.libs.utils.chart_tools.utc_time_string_for_javascript(date_string)¶
Convert 2014-01-16T12:07:23.322+03 to 2014-01-16T12:07:23.322+03:00
Cant use datetime.str[fp]time here since python 2.7’s %z is platform dependant - http://stackoverflow.com/questions/2609259/converting-string-t o-datetime-object-in-python
onadata.libs.utils.common_tools module¶
Common helper functions
- onadata.libs.utils.common_tools.cmp_to_key(mycmp)¶
Convert a cmp= function into a key= function
- onadata.libs.utils.common_tools.current_site_url(path, host)¶
Returns fully qualified URL (no trailing slash) for the current site. :param path :return: complete url
- onadata.libs.utils.common_tools.filename_from_disposition(content_disposition)¶
Gets a filename from the given content disposition header.
- onadata.libs.utils.common_tools.get_boolean_value(str_var, default=None)¶
Converts a string into boolean
- onadata.libs.utils.common_tools.get_choice_label(label, data_dictionary, language=None)¶
Return the label matching selected language or simply just the label.
- onadata.libs.utils.common_tools.get_choice_label_value(key, value, data_dictionary, language=None)¶
Return the label of a choice matching the value if the key xpath is a SELECT_ONE otherwise it returns the value unchanged.
- onadata.libs.utils.common_tools.get_response_content(response, decode=True)¶
Gets HTTP content for the given a HTTP response object.
Handles the case where a streaming_content is in the response.
- Parameters:
response – The response to extract content from.
decode – If true decode as utf-8, default True.
- onadata.libs.utils.common_tools.get_value_or_attachment_uri(key, value, row, data_dictionary, media_xpaths, attachment_list=None, show_choice_labels=False, language=None, host=None)¶
Gets either the attachment value or the attachment url :param key: used to retrieve survey element :param value: filename :param row: current records row :param data_dictionary: form structure :param include_images: boolean value to either inlcude images or not :param attachment_list: to be used incase row doesn’t have ATTACHMENTS key :return: value
- onadata.libs.utils.common_tools.json_stream(data, json_string)¶
Generator function to stream JSON data
- onadata.libs.utils.common_tools.merge_dicts(*dict_args)¶
Given any number of dicts, shallow copy and merge into a new dict, precedence goes to key value pairs in latter dicts.
- onadata.libs.utils.common_tools.report_exception(subject, info, exc_info=None)¶
Formats an exception then posts it to sentry and if not in debug or testing sends email to mail_admins.
- onadata.libs.utils.common_tools.retry(tries, delay=3, backoff=2)¶
- Adapted from code found here:
Retries a function or method until it returns True.
- Parameters:
delay – sets the initial delay in seconds, and backoff sets the
factor by which the delay should lengthen after each failure. :param backoff: must be greater than 1, or else it isn’t really a backoff. :param tries: must be at least 0, and delay greater than 0.
- onadata.libs.utils.common_tools.str_to_bool(str_var)¶
Return boolean True or False if string s represents a boolean value
- onadata.libs.utils.common_tools.track_task_progress(additions, total=None)¶
Updates the current export task with number of submission processed. Updates in batches of settings EXPORT_TASK_PROGRESS_UPDATE_BATCH defaults to 100. :param additions: :param total: :return:
onadata.libs.utils.country_field module¶
CountryField - provides a list of countries.
onadata.libs.utils.csv_builder module¶
CSV export utility functions.
- class onadata.libs.utils.csv_builder.AbstractDataFrameBuilder(username, id_string, filter_query=None, group_delimiter='/', split_select_multiples=True, binary_select_multiples=False, start=None, end=None, remove_group_name=False, xform=None, include_labels=False, include_labels_only=False, include_images=True, include_hxl=False, win_excel_utf8=False, total_records=None, index_tags=('[', ']'), value_select_multiples=False, show_choice_labels=True, include_reviews=False, language=None, host=None)¶
Bases:
object
Abstract Data Frame Builder class
- ADDITIONAL_COLUMNS = ['_id', '_uuid', '_submission_time', '_date_modified', '_tags', '_notes', '_version', '_duration', '_submitted_by', '_total_media', '_media_count', '_media_all_received']¶
- BINARY_SELECT_MULTIPLES = False¶
- IGNORED_COLUMNS = ['_xform_id_string', '_status', '_attachments', '_geolocation', '_bamboo_dataset_id', '_deleted_at', '_edited']¶
- VALUE_SELECT_MULTIPLES = False¶
Group functionality used by any DataFrameBuilder i.e. XLS, CSV and KML
- class onadata.libs.utils.csv_builder.CSVDataFrameBuilder(username, id_string, filter_query=None, group_delimiter='/', split_select_multiples=True, binary_select_multiples=False, start=None, end=None, remove_group_name=False, xform=None, include_labels=False, include_labels_only=False, include_images=False, include_hxl=False, win_excel_utf8=False, total_records=None, index_tags=('[', ']'), value_select_multiples=False, show_choice_labels=False, include_reviews=False, language=None, host=None)¶
Bases:
AbstractDataFrameBuilder
CSV data frame builder
- export_to(path, cursor, dataview=None)¶
Export a CSV formated to the given
path
.
- onadata.libs.utils.csv_builder.get_column_names_only(columns, data_dictionary, group_delimiter)¶
Return column names as a list.
- onadata.libs.utils.csv_builder.get_labels_from_columns(columns, data_dictionary, group_delimiter, language=None)¶
Return
column
labels
- onadata.libs.utils.csv_builder.get_prefix_from_xpath(xpath)¶
Returns xpath prefix.
- onadata.libs.utils.csv_builder.remove_dups_from_list_maintain_order(lst)¶
Removes duplicates from a list and still maintains the order.
- onadata.libs.utils.csv_builder.write_to_csv(path, rows, columns, columns_with_hxl=None, remove_group_name=False, data_dictionary=None, group_delimiter='/', include_labels=False, include_labels_only=False, include_hxl=False, win_excel_utf8=False, total_records=None, index_tags=('[', ']'), language=None)¶
Writes
rows
to a file in CSV format.
onadata.libs.utils.csv_import module¶
CSV data import module.
- onadata.libs.utils.csv_import.dict2xmlsubmission(submission_dict, xform, instance_id, submission_date)¶
Creates and xml submission from an appropriate dict (& other data)
- Parameters:
submission_dict (dict) – A dict containing form submission data.
xfrom (onadata.apps.logger.models.XForm) – The submission’s XForm.
instance_id (string) – The submission/instance uuid.
submission_date (string) – An isoformatted datetime string.
- Returns:
An xml submission string
- Return type:
string
- onadata.libs.utils.csv_import.dict_merge(part_a, part_b)¶
Returns a merger of two dicts a and b
credits: https://www.xormedia.com/recursively-merge-dictionaries-in-python
- onadata.libs.utils.csv_import.dict_pathkeys_to_nested_dicts(dictionary)¶
Turns a flat dict to a nested dict
Takes a dict with pathkeys or “slash-namespaced” keys and inflates them into nested dictionaries i.e:- d[‘/path/key/123’] -> d[‘path’][‘key’][‘123’]
- onadata.libs.utils.csv_import.failed_import(rollback_uuids, xform, exception, status_message)¶
Report a failed import. :param rollback_uuids: The rollback UUIDs :param xform: The XForm that failed to import to :param exception: The exception object :return: The async_status result
- onadata.libs.utils.csv_import.flatten_split_select_multiples(row: Dict[str, Any], select_multiples: List[str]) dict ¶
Flattens a select_multiple question that was previously split into different choice columns into one column
- onadata.libs.utils.csv_import.get_async_csv_submission_status(job_uuid)¶
Gets CSV Submision progress or result Can be used to pol long running submissions :param str job_uuid: The submission job uuid returned by _submit_csv.delay :return: Dict with import progress info (insertions & total) :rtype: Dict
- onadata.libs.utils.csv_import.get_columns_by_type(type_list, form_json)¶
Returns a column that match types passed within the field_list
- Parameters:
type_list (list) – A list containing strings that represent
XLS field types :param dict form_json: A dict representing the contents of a form :return: A list containing the column names that store values passed within the type_list :rtype: list
- onadata.libs.utils.csv_import.get_submission_meta_dict(xform, instance_id)¶
Generates metadata for our submission
Checks if instance_id belongs to an existing submission. If it does, it’s considered an edit and its uuid gets deprecated. In either case, a new one is generated and assigned.
- Parameters:
xform (onadata.apps.logger.models.XForm) – The submission’s XForm.
instance_id (string) – The submission/instance uuid.
- Returns:
The metadata dict
- Return type:
- onadata.libs.utils.csv_import.submission_xls_to_csv(xls_file)¶
Convert a submission xls file to submissions csv file
- Parameters:
xls_file – submissions xls file
- Returns:
csv_file
- onadata.libs.utils.csv_import.submit_csv(username, xform, csv_file, overwrite=False)¶
Imports CSV data to an existing form
Takes a csv formatted file or string containing rows of submission/instance and converts those to xml submissions and finally submits them by calling
onadata.libs.utils.logger_tools.safe_create_instance()
- onadata.libs.utils.csv_import.validate_column(row, columns, constraint_check)¶
Validates columns within a row making sure data constraints are not broken
Takes a list of column headers to validate and a function of which is used to validate the data is valid.
- Parameters:
row (Dict) – The row of which the columns values should be validated
columns (List) – A list of headers to be validated within the row.
constraint_check (func) – A function used to validate column data
- Returns:
Returns a tuple containing the validity status of the rows
data and the validated_data if successful else it contains the invalid_data
- onadata.libs.utils.csv_import.validate_csv_file(csv_file, xform)¶
Validates a CSV File
Takes a CSV Formatted file or sring containing rows of submission data and validates that the file is valid enough to be processed.
- Parameters:
csv_file ((str or file)) – A CSV formatted file or string with
submission rows :param onadata.apps.logger.models.XForm xform: The submission’s XForm :rtype: Dict :returns: A dict containing the validity of the CSV file as well as additional columns(additional_col) if any when successful else it returns an error message(error_msg)
- onadata.libs.utils.csv_import.validate_row(row, columns)¶
Validate row of data making sure column constraints are enforced
Takes a csv row containing data from a submission and validates the data making sure data types are enforced.
- Parameters:
row (Dict) – The csv row
columns (Dict) – A dict containing the column headers to be validated
and the function that should check that the columns constraint is not broken :return: Returns a tuple containing the validated row and errors found within the row if any :rtype: Tuple
onadata.libs.utils.csv_reader module¶
CsvReader class module.
- class onadata.libs.utils.csv_reader.CsvReader(path)¶
Bases:
object
Typical usage:
csv_reader = CsvReader(path) for d in csv_reader.iter_dicts(): Variable.objects.create(**d)
- close()¶
Closes the file handle.
- iter_dicts()¶
Iterate over CSV rows as dict items.
- next()¶
A CsvReader object is iterable (since we have defined __iter__ and next methods. Each iteration of this object returns a row of data.
- open(path)¶
Opens a file handle sets a CSV reader.
onadata.libs.utils.decorators module¶
decorators module
- onadata.libs.utils.decorators.check_obj(f)¶
Checks if the first argument is truthy and then calls the underlying function.
- onadata.libs.utils.decorators.is_owner(view_func)¶
Redirects to login if not owner.
onadata.libs.utils.dict_tools module¶
Dict utility functions module.
- onadata.libs.utils.dict_tools.csv_dict_to_nested_dict(csv_dict, select_multiples=None)¶
Converts a CSV dict to nested dicts.
- onadata.libs.utils.dict_tools.dict_lists2strings(adict)¶
Convert lists in a dict to joined strings.
- Parameters:
d – The dict to convert.
- Returns:
The converted dict.
- onadata.libs.utils.dict_tools.dict_paths2dict(adict)¶
Turns a dict with ‘/’ in keys to a nested dict.
- onadata.libs.utils.dict_tools.floip_response_headers_dict(data, xform_headers)¶
Returns a dict from matching xform headers and floip responses.
- onadata.libs.utils.dict_tools.get_values_matching_key(doc, key)¶
Returns iterator of values in ‘doc’ with the matching ‘key’.
- onadata.libs.utils.dict_tools.list_to_dict(items, value)¶
Converts a list into a dict.
- onadata.libs.utils.dict_tools.merge_list_of_dicts(list_of_dicts, override_keys: list | None = None)¶
Merges a list of dicts to return one dict.
- onadata.libs.utils.dict_tools.query_list_to_dict(query_list_str)¶
Returns a ‘label’ and ‘text’ from a Rapidpro values JSON string as a dict.
- onadata.libs.utils.dict_tools.remove_indices_from_dict(obj)¶
Removes indices from a obj dict.
onadata.libs.utils.export_builder module¶
ExportBuilder
- class onadata.libs.utils.export_builder.ExportBuilder¶
Bases:
object
Utility class for generating multiple formats of data export to file.
- BINARY_SELECT_MULTIPLES = False¶
- CONVERT_FUNCS = {'date': <function string_to_date_with_xls_validation>, 'dateTime': <function ExportBuilder.<lambda>>, 'decimal': <class 'float'>, 'int': <class 'int'>}¶
- EXTRA_FIELDS = ['_id', '_uuid', '_submission_time', '_index', '_parent_table_name', '_parent_index', '_tags', '_notes', '_version', '_duration', '_submitted_by']¶
- GROUP_DELIMITER = '/'¶
- GROUP_DELIMITERS = ['/', '.']¶
- GROUP_DELIMITER_DOT = '.'¶
- GROUP_DELIMITER_SLASH = '/'¶
- IGNORED_COLUMNS = ['_xform_id_string', '_status', '_attachments', '_geolocation', '_bamboo_dataset_id', '_deleted_at']¶
- INCLUDE_HXL = False¶
- INCLUDE_IMAGES = True¶
- INCLUDE_LABELS = False¶
- INCLUDE_LABELS_ONLY = False¶
- INCLUDE_REVIEWS = False¶
- REPEAT_INDEX_TAGS = ('[', ']')¶
- SHOW_CHOICE_LABELS = False¶
- SPLIT_SELECT_MULTIPLES = True¶
- TRUNCATE_GROUP_TITLE = False¶
- TYPES_TO_CONVERT = ['int', 'decimal', 'date']¶
- VALUE_SELECT_MULTIPLES = False¶
- XLS_SHEET_NAME_MAX_CHARS = 31¶
- classmethod convert_type(value, data_type)¶
Convert data to its native type e.g. string ‘1’ to int 1 @param value: the string value to convert @param data_type: the native data type to convert to @return: the converted value
- classmethod decode_mongo_encoded_fields(row, encoded_fields)¶
Update encoded fields with their corresponding xpath
- classmethod format_field_title(abbreviated_xpath, field_delimiter, data_dictionary, remove_group_name=False, gps_extra_field=False)¶
Format the field title.
- get_choice_label_from_dict(label)¶
Returns the choice label for the default language
If a label for the target language is blank then the default language is used
- get_default_language(languages)¶
Return the default languange of the XForm.
- get_fields(dataview, section, key)¶
Return list of element value with the key in section[‘elements’].
- classmethod get_valid_sheet_name(desired_name, existing_names)¶
Returns a valid sheet_name based on the desired names
- language = None¶
- pre_process_row(row, section)¶
Split select multiples, gps and decode . and $
- section_by_name(name)¶
Return section by the given
name
.
- set_survey(survey, xform=None, include_reviews=False)¶
Set’s the XForm XML
survey
instance.
- classmethod split_gps_components(row, gps_fields)¶
Splits GPS components into their own fields.
- classmethod split_select_multiples(row, select_multiples, select_values=False, binary_select_multiples=False, show_choice_labels=False, data_dictionary=None, language=None)¶
Split select multiple choices in a submission to individual columns.
- Parameters:
row – single submission dict
select_multiples – list of XPATHs and choices of select multiple questions.
binary_select_multiples – if True the value of the split columns will be 1 when the choice has been selected otherwise it will be 0.
select_values – the value of the split columns will be the name/value of the choice when selected otherwise blank/None.
show_choice_labels – Show a choice label instead of the value/True/False/1/0.
data_dictionary – A DataDictionary/XForm object
language – specific language as defined in the XLSForm.
- Returns:
the row dict with select multiples choice as fields in the row
- to_flat_csv_export(path, data, username, id_string, filter_query, **kwargs)¶
Generates a flattened CSV file for submitted data.
- to_xlsx_export(path, data, *args, **kwargs)¶
Export data to a spreadsheet document.
- to_zipped_csv(path, data, *args, **kwargs)¶
Export CSV formatted files from
data
and zip the files.
- to_zipped_sav(path, data, *args, **kwargs)¶
Generates the SPSS zipped file format export.
- url = None¶
- onadata.libs.utils.export_builder.decode_mongo_encoded_section_names(data)¶
Recursively decode mongo keys.
- Parameters:
data – A dictionary to decode.
- onadata.libs.utils.export_builder.dict_to_joined_export(data, index, indices, name, survey, row, host, media_xpaths=None)¶
Converts a dict into one or more tabular datasets :param data: current record which can be changed or updated :param index: keeps count of record number :param indices: a dictionary storing list values if data is a dict :param name: the name of the survey :param survey: the survey :param row: current record that remains unchanged on this function’s recall
- onadata.libs.utils.export_builder.encode_if_str(row, key, encode_dates=False, sav_writer=None)¶
Encode a string value in
row[key]
.
- onadata.libs.utils.export_builder.get_data_dictionary_from_survey(survey)¶
Creates a DataDictionary instance from an XML survey instance.
- onadata.libs.utils.export_builder.is_all_numeric(items)¶
Check if all items on the list are numeric, zero padded numbers will not be considered as numeric.
- Parameters:
items – list of values to be checked
- Return boolean:
- onadata.libs.utils.export_builder.string_to_date_with_xls_validation(date_str)¶
Try to convert a string to a date object.
- Parameters:
date_str – string to convert
- Returns:
object if converted, otherwise date string
onadata.libs.utils.export_tools module¶
Export tools
- onadata.libs.utils.export_tools.check_pending_export(xform, export_type, options, minutes=5)¶
Check for pending export done within a specific period of time and returns the export :param xform: :param export_type: :param options: :param minutes :return:
- onadata.libs.utils.export_tools.clean_keys_of_slashes(record)¶
Replaces the slashes found in a dataset keys with underscores :param record: list containing a couple of dictionaries :return: record with keys without slashes
- onadata.libs.utils.export_tools.create_export_object(xform, export_type, options)¶
Return an export object that has not been saved to the database.
- onadata.libs.utils.export_tools.generate_attachments_zip_export(export_type, username, id_string, export_id=None, options=None, xform=None)¶
Generates zip export of attachments.
param: export_type params: username: logged in username params: id_string: xform id_string params: export_id: ID of export object associated with the request param: options: additional parameters required for the lookup.
ext: File extension of the generated export
- onadata.libs.utils.export_tools.generate_external_export(export_type, username, id_string, export_id=None, options=None, xform=None)¶
Generates external export using ONA data through an external service.
param: export_type params: username: logged in username params: id_string: xform id_string params: export_id: ID of export object associated with the request param: options: additional parameters required for the lookup.
data_id: instance id query: filter_query for custom queries meta: metadata associated with external export token: authentication key required by external service
- onadata.libs.utils.export_tools.generate_geojson_export(export_type, username, id_string, metadata=None, export_id=None, options=None, xform=None)¶
Generates Linked Geojson export
- Parameters:
export_type – type of export
username – logged in username
id_string – xform id_string
export_id – ID of export object associated with the request
options – additional parameters required for the lookup.
ext – File extension of the generated export
- onadata.libs.utils.export_tools.generate_kml_export(export_type, username, id_string, export_id=None, options=None, xform=None)¶
Generates kml export for geographical data
- Parameters:
export_type – type of export
username – logged in username
id_string – xform id_string
export_id – ID of export object associated with the request
options – additional parameters required for the lookup.
extension – File extension of the generated export
- onadata.libs.utils.export_tools.generate_osm_export(export_type, username, id_string, export_id=None, options=None, xform=None)¶
Generates osm export for OpenStreetMap data
- Parameters:
export_type – type of export
username – logged in username
id_string – xform id_string
export_id – ID of export object associated with the request
options – additional parameters required for the lookup.
ext – File extension of the generated export
- onadata.libs.utils.export_tools.get_export_options(options)¶
Returns expirt options as defined in Export.EXPORT_OPTION_FIELDS from a list of provided options to be saved with each Export object.
- onadata.libs.utils.export_tools.get_or_create_export(export_id, xform, export_type, options)¶
Returns an existing export object or creates a new one with the given options.
- onadata.libs.utils.export_tools.get_or_create_export_object(export_id, options, xform, export_type)¶
Get or create export object.
- Parameters:
export_id – Export ID
options – Options to convert to export options
xform – XForm to export
export_type – The type of export
- Returns:
A new or found export object
- onadata.libs.utils.export_tools.get_osm_data_kwargs(xform)¶
Return kwargs for OsmData queryset for given xform
- onadata.libs.utils.export_tools.get_repeat_index_tags(index_tags)¶
Gets a comma separated string index_tags
Retuns a tuple of two strings with SUPPORTED_INDEX_TAGS,
- onadata.libs.utils.export_tools.increment_index_in_filename(filename)¶
filename should be in the form file.ext or file-2.ext - we check for the dash and index and increment appropriately
- onadata.libs.utils.export_tools.kml_export_data(id_string, user, xform=None)¶
KML export data from form submissions.
- onadata.libs.utils.export_tools.md5hash(string)¶
Return the MD5 hex digest of the given string.
- onadata.libs.utils.export_tools.newest_export_for(xform, export_type, options)¶
Retrieve the latest export given the following arguments:
param: xform param: export_type param: options: additional parameters required for the lookup.
remove_group_name: boolean flag group_delimiter: “/” or “.” with “/” as the default split_select_multiples: boolean flag binary_select_multiples: boolean flag index_tag: (‘[’, ‘]’) or (‘_’, ‘_’)
- onadata.libs.utils.export_tools.parse_request_export_options(params)¶
Parse export options in the request object into values returned in a list. The list represents a boolean for whether the group name should be removed, the group delimiter, and a boolean for whether select multiples should be split.
- onadata.libs.utils.export_tools.should_create_new_export(xform, export_type, options, request=None)¶
Function that determines whether to create a new export. param: xform param: export_type param: options: additional parameters required for the lookup.
remove_group_name: boolean flag group_delimiter: “/” or “.” with “/” as the default split_select_multiples: boolean flag binary_select_multiples: boolean flag index_tag: (‘[’, ‘]’) or (‘_’, ‘_’)
params: request: Get params are used to determine if new export is required
- onadata.libs.utils.export_tools.upload_template_for_external_export(server, file_obj)¶
Uploads an Excel template to the XLSReport server.
Returns the status code with the server response.
- onadata.libs.utils.export_tools.write_temp_file_to_path(suffix, content, file_path)¶
Write a temp file and return the name of the file. :param suffix: The file suffix :param content: The content to write :param file_path: The path to write the temp file to :return: The filename written to
onadata.libs.utils.google module¶
Google utility functions.
onadata.libs.utils.gravatar module¶
Gravatar utils module
- onadata.libs.utils.gravatar.email_md5(user)¶
Returns the hash of an email for the user
- onadata.libs.utils.gravatar.get_gravatar_img_link(user)¶
Returns the Gravatar image URL
- onadata.libs.utils.gravatar.gravatar_exists(user)¶
Checks if the Gravatar URL exists
onadata.libs.utils.image_tools module¶
Image utility functions module.
- onadata.libs.utils.image_tools.flat(*nums)¶
Build a tuple of ints from float or integer arguments. Useful because PIL crop and resize require integer points. source: https://gist.github.com/16a01455
- onadata.libs.utils.image_tools.generate_aws_media_url(file_path: str, content_disposition: str, expiration: int = 3600)¶
Generate S3 URL.
- onadata.libs.utils.image_tools.generate_media_download_url(obj, expiration: int = 3600)¶
Returns a HTTP response of a media object or a redirect to the image URL for S3 and Azure storage objects.
- onadata.libs.utils.image_tools.generate_media_url_with_sas(file_path: str, expiration: int = 3600)¶
Generate Azure storage URL.
- onadata.libs.utils.image_tools.get_dimensions(size, longest_side)¶
Return integer tuple of width and height given size and longest_side length.
- onadata.libs.utils.image_tools.image_url(attachment, suffix)¶
Return url of an image given size(@param suffix) e.g large, medium, small, or generate required thumbnail
- onadata.libs.utils.image_tools.is_azure_storage()¶
Checks if azure storage is in use
- onadata.libs.utils.image_tools.resize(filename, extension)¶
Resize an image into multiple sizes.
- onadata.libs.utils.image_tools.resize_local_env(filename, extension)¶
Resize images in a local environment.
onadata.libs.utils.log module¶
Log utility functions and classes.
- class onadata.libs.utils.log.AuditLogHandler(model='')¶
Bases:
Handler
Audit logging handler class.
- emit(record)¶
Do whatever it takes to actually log the specified logging record.
This version is intended to be implemented by subclasses and so raises a NotImplementedError.
- get_model(name)¶
Import and return the model under the given
name
.
- onadata.libs.utils.log.audit_log(action, request_user, account_user, message, audit, request, level=10)¶
Create a log message based on these params
@param action: Action performed e.g. form-deleted @param request_username: User performing the action @param account_username: The formhub account the action was performed on @param message: The message to be displayed on the log @param level: log level @param audit: a dict of key/values of other info pertaining to the action e.g. form’s id_string, submission uuid @return: None
onadata.libs.utils.logger_tools module¶
logger_tools - Logger app utility functions.
- class onadata.libs.utils.logger_tools.BaseOpenRosaResponse(*args, **kwargs)¶
Bases:
HttpResponse
The base HTTP response class with OpenRosa headers.
- status_code = 201¶
- class onadata.libs.utils.logger_tools.OpenRosaNotAuthenticated(*args, **kwargs)¶
Bases:
Response
An HTTP response class with OpenRosa headers for the Not Authenticated response.
- status_code = 401¶
- class onadata.libs.utils.logger_tools.OpenRosaResponse(*args, **kwargs)¶
Bases:
BaseOpenRosaResponse
An HTTP response class with OpenRosa headers for the created response.
- status_code = 201¶
- class onadata.libs.utils.logger_tools.OpenRosaResponseBadRequest(*args, **kwargs)¶
Bases:
OpenRosaResponse
An HTTP response class with OpenRosa headers for the Bad Request response.
- status_code = 400¶
- class onadata.libs.utils.logger_tools.OpenRosaResponseForbidden(*args, **kwargs)¶
Bases:
OpenRosaResponse
An HTTP response class with OpenRosa headers for the Forbidden response.
- status_code = 403¶
- class onadata.libs.utils.logger_tools.OpenRosaResponseNotAllowed(*args, **kwargs)¶
Bases:
OpenRosaResponse
An HTTP response class with OpenRosa headers for the Not Allowed response.
- status_code = 405¶
- class onadata.libs.utils.logger_tools.OpenRosaResponseNotFound(*args, **kwargs)¶
Bases:
OpenRosaResponse
An HTTP response class with OpenRosa headers for the Not Found response.
- status_code = 404¶
- class onadata.libs.utils.logger_tools.PublishXForm(xml_file, user)¶
Bases:
object
A class to publish an XML XForm file.
- publish_xform()¶
Publish an XForm XML file.
- onadata.libs.utils.logger_tools.check_edit_submission_permissions(request_user, xform)¶
Checks edit submission permissions.
- onadata.libs.utils.logger_tools.check_submission_encryption(xform: XForm, xml: bytes) NoReturn ¶
Check that the submission is encrypted or unencrypted depending on the encryption status of an XForm.
The submission is invalid if the XForm’s encryption status is different from the submissions
- onadata.libs.utils.logger_tools.check_submission_permissions(request, xform)¶
Check that permission is required and the request user has permission.
- The user does no have permissions iff:
the user is authed,
either the profile or the form require auth,
the xform user is not submitting.
Since we have a username, the Instance creation logic will handle checking for the forms existence by its id_string.
- Returns:
None.
- Raises:
PermissionDenied based on the above criteria.
- onadata.libs.utils.logger_tools.create_instance(username, xml_file, media_files, status='submitted_via_web', uuid=None, date_created_override=None, request=None)¶
I used to check if this file had been submitted already, I’ve taken this out because it was too slow. Now we’re going to create a way for an admin to mark duplicate instances. This should simplify things a bit. Submission cases: * If there is a username and no uuid, submitting an old ODK form. * If there is a username and a uuid, submitting a new ODK form.
- onadata.libs.utils.logger_tools.create_xform_version(xform: XForm, user: User) XFormVersion ¶
Creates an XFormVersion object for the passed in XForm
- onadata.libs.utils.logger_tools.dict2xform(jsform, form_id, root=None, username=None, gen_uuid=False)¶
Converts a dictionary containing submission data into an XML Submission for the appropriate form.
- Parameters:
(dict) (jsform) – A python dictionary object containing the submission data
XForm) (form_id (str or) – An XForm object or a string value representing the forms id_string
(str) (root) – An optional string that should be used as the root nodes name. Defaults to None
- Param:
username (str): An optional string representing a users username. Used alongside the form_id to locate the XForm object the user is trying to submit data too. Defaults to None
- Returns:
Returns a string containing the Submission XML
- Return type:
- onadata.libs.utils.logger_tools.generate_content_disposition_header(name, extension, show_date=True)¶
Returns the a Content-Description header formatting string,
- onadata.libs.utils.logger_tools.get_filtered_instances(*args, **kwargs)¶
Get filtered instances - mainly to allow mocking in tests
- onadata.libs.utils.logger_tools.get_first_record(queryset)¶
Returns the first item in a queryset sorted by id.
- onadata.libs.utils.logger_tools.get_uuid_from_submission(xml)¶
Extracts and returns the UUID from a submission XML.
- onadata.libs.utils.logger_tools.get_xform_from_submission(xml, username, uuid=None, request=None)¶
Gets the submissions target XForm.
Retrieves the target XForm by either utilizing the uuid param or the uuid retrievable from the xml or the id_string retrievable from the XML. Only returns form if request_user has permission to submit.
- Parameters:
Default: None :param (django.http.request) request: Request object. Default: None
- onadata.libs.utils.logger_tools.inject_instanceid(xml_str, uuid)¶
- onadata.libs.utils.logger_tools.publish_form(callback)¶
Calls the callback function to publish a XLSForm and returns appropriate message depending on exception throw during publishing of a XLSForm.
- onadata.libs.utils.logger_tools.remove_metadata_fields(data)¶
Clean out unneccessary metadata fields
- onadata.libs.utils.logger_tools.response_with_mimetype_and_name(mimetype, name, extension=None, show_date=True, file_path=None, use_local_filesystem=False, full_mime=False)¶
Returns a HttpResponse with Content-Disposition header set
Triggers a download on the browser.
- onadata.libs.utils.logger_tools.safe_create_instance(username, xml_file, media_files, uuid, request, instance_status: str = 'submitted_via_web')¶
Create an instance and catch exceptions.
- Returns:
A list [error, instance] where error is None if there was no error.
- onadata.libs.utils.logger_tools.save_attachments(xform, instance, media_files, remove_deleted_media=False)¶
Saves attachments for the given instance/submission.
- onadata.libs.utils.logger_tools.save_submission(xform, xml, media_files, new_uuid, submitted_by, status, date_created_override, checksum, request=None)¶
Persist a submission into the ParsedInstance model.
- onadata.libs.utils.logger_tools.set_default_openrosa_headers(response)¶
Sets the default OpenRosa headers into a
response
object.
- onadata.libs.utils.logger_tools.store_temp_file(data)¶
Creates a temporary file with the
data
and returns it.
- onadata.libs.utils.logger_tools.update_attachment_tracking(instance)¶
Takes an Instance object and updates attachment tracking fields
onadata.libs.utils.middleware module¶
Custom middleware classes.
- class onadata.libs.utils.middleware.BaseMiddleware(get_response)¶
Bases:
object
BaseMiddleware - The base middleware class.
- class onadata.libs.utils.middleware.ExceptionLoggingMiddleware(get_response)¶
Bases:
object
The exception logging middleware class - prints the exception traceback.
- process_exception(request, exception)¶
Prints the exception traceback.
- class onadata.libs.utils.middleware.HTTPResponseNotAllowedMiddleware(get_response)¶
Bases:
object
The HTTP Not Allowed middleware class - renders the 405.html template.
- class onadata.libs.utils.middleware.LocaleMiddlewareWithTweaks(get_response=None)¶
Bases:
LocaleMiddleware
- Overrides LocaleMiddleware from django with:
Khmer km language code in Accept-Language is rewritten to km-kh
- process_request(request)¶
- class onadata.libs.utils.middleware.OperationalErrorMiddleware(get_response)¶
Bases:
BaseMiddleware
Captures requests returning 500 status code. Then retry it against master database.
- process_exception(request, exception)¶
Handle retrying OperatuonalError exceptions.
onadata.libs.utils.model_tools module¶
Model utility functions.
- onadata.libs.utils.model_tools.get_columns_with_hxl(survey_elements)¶
Returns a dictionary whose keys are xform field names and values are instance::hxl values set on the xform :param include_hxl - boolean value :param survey_elements - survey elements of an xform return dictionary or None
- onadata.libs.utils.model_tools.queryset_iterator(queryset, chunksize=100)¶
Iterate over a Django Queryset.
This method loads a maximum of chunksize (default: 100) rows in its memory at the same time while django normally would load all rows in its memory. Using the iterator() method only causes it to not preload all the classes.
See https://docs.djangoproject.com/en/2.1/ref/models/querysets/#iterator
- onadata.libs.utils.model_tools.set_uuid(obj)¶
Only give an object a new UUID if it does not have one.
onadata.libs.utils.mongo module¶
Utility functions for MongoDB operations.
onadata.libs.utils.numeric module¶
The int_or_parse_error utility function.
- onadata.libs.utils.numeric.int_or_parse_error(value, error_string)¶
If value is not an int raise a parse error with error_string, which is a format string that takes one argument, the value.
onadata.libs.utils.osm module¶
OSM utility module.
- onadata.libs.utils.osm.get_combined_osm(osm_list)¶
Combine osm xml form list of OsmData objects
- onadata.libs.utils.osm.osm_flat_dict(instance_id)¶
Flat dict of OSM tags for the specified submission.
Each key starts with ‘osm_*’.
- onadata.libs.utils.osm.parse_osm(osm_xml, include_osm_id=False)¶
Parses OSM XML and return a list of ways or nodes.
- onadata.libs.utils.osm.parse_osm_nodes(osm_xml, include_osm_id=False)¶
Converts an OSM XMl to a list of GEOSGeometry objects
- onadata.libs.utils.osm.parse_osm_tags(node, include_osm_id=False)¶
Retrieves all the tags from a osm xml node
- onadata.libs.utils.osm.parse_osm_ways(osm_xml, include_osm_id=False)¶
Converts an OSM XMl to a list of GEOSGeometry objects
- onadata.libs.utils.osm.save_osm_data(instance_id)¶
Includes the OSM data in the specified submission json data.
onadata.libs.utils.profiler module¶
onadata.libs.utils.project_utils module¶
project_utils module - apply project permissions to a form.
- exception onadata.libs.utils.project_utils.ExternalServiceRequestError¶
Bases:
Exception
Custom Exception class for External service requests i.e Formbuilder
- onadata.libs.utils.project_utils.assign_change_asset_permission(service_url: str, asset_id: str, usernames: List[str], session: Session) Response ¶
Bulk assigns the change_asset permission to a group of users on KPI
- onadata.libs.utils.project_utils.get_project_users(project)¶
Return project users with the role assigned to them.
- onadata.libs.utils.project_utils.propagate_project_permissions(project: Project, headers: dict | None = None, use_asset_owner_auth: bool = True) None ¶
Propagates Project permissions to external services(KPI)
- onadata.libs.utils.project_utils.retrieve_asset_permissions(service_url: str, asset_id: str, session: Session) Dict[str, List[str]] ¶
Retrieves the currently assigned asset permissions for each user on KPI
- onadata.libs.utils.project_utils.set_project_perms_to_xform(xform, project)¶
Apply project permissions to a form, this usually happens when a new form is being published or it is being moved to a new project.
onadata.libs.utils.qrcode module¶
QR code utility function.
- onadata.libs.utils.qrcode.generate_qrcode(message)¶
Generate a QRCode, settings options and output.
onadata.libs.utils.quick_converter module¶
The QuickConverter form class - publishes XLSForms.
- class onadata.libs.utils.quick_converter.QuickConverter(data=None, files=None, auto_id='id_%s', prefix=None, initial=None, error_class=<class 'django.forms.utils.ErrorList'>, label_suffix=None, empty_permitted=False, field_order=None, use_required_attribute=None, renderer=None)¶
Bases:
Form
The QuickConverter form - publishes XLSForms.
- base_fields = {'xls_file': <django.forms.fields.FileField object>}¶
- declared_fields = {'xls_file': <django.forms.fields.FileField object>}¶
- property media¶
Return all media required to render the widgets on this form.
- publish(user)¶
Create and return a DataDictionary object.
onadata.libs.utils.string module¶
String utility function str2bool - converts yes, true, t, 1 to True else returns the argument value v.
- onadata.libs.utils.string.str2bool(v)¶
String utility function str2bool - converts “yes”, “true”, “t”, “1” to True else returns the argument value v.
onadata.libs.utils.timing module¶
Custom date utility functions.
- onadata.libs.utils.timing.calculate_duration(start_time, end_time)¶
This function calculates duration when given start and end times. An empty string is returned if either of the time formats does not match ‘_format’ format else, the duration is returned
- onadata.libs.utils.timing.get_date(_object=None)¶
Returns a date formatted string of the date_modified of the given
_object
.
- onadata.libs.utils.timing.get_header_date_format(date_modified)¶
Returns
date_modified
as a string with the format %a, %d %b %Y %H:%M:%S GMT
- onadata.libs.utils.timing.last_modified_header(last_modified_date)¶
Returns a dictionary with the ‘Last-Modified’ key and value.
onadata.libs.utils.user_auth module¶
User authentication utility functions.
- class onadata.libs.utils.user_auth.HttpResponseNotAuthorized(*args, **kwargs)¶
Bases:
HttpResponse
HttpResponse that sets basic authentication prompt.
- status_code = 401¶
- onadata.libs.utils.user_auth.add_cors_headers(response)¶
Add CORS headers to the HttpResponse
response
instance.
- onadata.libs.utils.user_auth.basic_http_auth(func)¶
A basic authentication decorator.
- onadata.libs.utils.user_auth.check_and_set_form_by_id(pk, request)¶
Checks xform by
pk
and returns an xform ifrequest.user
has permission.
- onadata.libs.utils.user_auth.check_and_set_form_by_id_string(username, id_string, request)¶
Checks xform by
id_string
and returns an xform ifrequest.user
has permission.
- onadata.libs.utils.user_auth.check_and_set_user(request, username)¶
Returns a User object or a string to redirect.
- onadata.libs.utils.user_auth.check_and_set_user_and_form(username, id_string, request)¶
Checks and returns an xform and owner if
request.user
has permission.
- onadata.libs.utils.user_auth.get_user_default_project(user)¶
Return’s the
user
’s default project, creates it if it does not exist.’
- onadata.libs.utils.user_auth.get_xform_and_perms(username, id_string, request)¶
Returns the xform with the matching
id_string
, and the permissions therequest.user
has.
- onadata.libs.utils.user_auth.get_xform_users_with_perms(xform)¶
Similar to django-guadian’s get_users_with_perms here the query makes use of the xformuserobjectpermission_set to return a dictionary of users with a list of permissions to the XForm object. The query in use is not as expensive as the one in use with get_users_with_perms
- onadata.libs.utils.user_auth.has_edit_permission(xform, owner, request, shared=False)¶
Checks if the
request.user
has edit permissions to thexform
.
- onadata.libs.utils.user_auth.has_permission(xform, owner, request, shared=False)¶
Checks if the
request.user
has the necessary permissions to anxform
.
- onadata.libs.utils.user_auth.helper_auth_helper(request)¶
Authenticates a user via basic authentication.
- onadata.libs.utils.user_auth.http_auth_string(username, password)¶
Return a basic authentication string with username and password.
- onadata.libs.utils.user_auth.invalidate_and_regen_tokens(user)¶
Invalidates a users Access and Temp tokens and generates new ones
- onadata.libs.utils.user_auth.set_api_permissions_for_user(user)¶
Sets the permissions to allow a
user
to access the APU.
- onadata.libs.utils.user_auth.set_profile_data(data, content_user)¶
Set user profile data.
onadata.libs.utils.viewer_tools module¶
Utility functions for data views.
- onadata.libs.utils.viewer_tools.create_attachments_zipfile(attachments, zip_file)¶
Return a zip file with submission attachments.
- Parameters:
attachments – an Attachments queryset.
zip_file – a file object, more likely a NamedTemporaryFile() object.
- onadata.libs.utils.viewer_tools.export_def_from_filename(filename)¶
Return file extension and mimetype from filename.
- onadata.libs.utils.viewer_tools.generate_enketo_form_defaults(xform, **kwargs)¶
Return Enketo default options for preloading data into a web form.
- onadata.libs.utils.viewer_tools.get_client_ip(request)¶
Return an IP from HTTP_X_FORWARDED_FOR or REMOTE_ADDR request headers.
arguments: request – HttpRequest object.
- onadata.libs.utils.viewer_tools.get_enketo_urls(form_url, id_string, instance_xml=None, instance_id=None, return_url=None, **kwargs) Dict[str, str] ¶
Return Enketo URLs.
- onadata.libs.utils.viewer_tools.get_form(kwargs)¶
Return XForm object by applying kwargs on an XForm queryset.
- onadata.libs.utils.viewer_tools.get_form_url(request, username=None, protocol='https', preview=False, xform_pk=None, generate_consistent_urls=False)¶
Return a form list url endpoint to be used to make a request to Enketo.
For example, it will return https://example.com and Enketo will know to look for the form list at https://example.com/formList. If a username is provided then Enketo will request the form list from https://example.com/[username]/formList. Same applies for preview if preview is True and also to a single form when xform_pk is provided.
- onadata.libs.utils.viewer_tools.get_path(path, suffix)¶
Apply the suffix to the path.
- onadata.libs.utils.viewer_tools.handle_enketo_error(response)¶
Handle enketo error response.
- onadata.libs.utils.viewer_tools.image_urls(instance)¶
Return image urls of all image attachments of the submission instance.
arguments: instance – Instance submission object.
- onadata.libs.utils.viewer_tools.image_urls_for_form(xform)¶
Return image urls of all image attachments of the xform.
- onadata.libs.utils.viewer_tools.parse_xform_instance(xml_str)¶
‘xml_str’ is a str object holding the XML of an XForm instance.
Return a python object representation of this XML file.