Importing data into Insight - file formats

Importing into Insight

Publishing data to Community Insight may make it public. Please revisit your Insight Terms of Use for more clarification.

All copyright laws apply here. If you have rights to a dataset, you can import it to Community Insight. If you aren't sure, first contact the owner of the data to check. That person may need permissions from authors or other publishers before it can be placed in a public data sharing tool such as Community Insight. You may attach standard or custom intellectual property rights on the dataset during the import.

Use the import button found at the top of every page to begin the four-step import process. It is helpful, however, to first consider whether your dataset file is well structured.

Data Format and Style

Gather your data and metadata and understand the acceptable formats for Community Insight: spreadsheets, ESRI shape files, GeoTIFF raster imagery, KML, or GPX.

General Guidance

The following guidelines should be used while preparing any dataset for import to Community Insight, regardless of format:

If you are uploading multiple files that comprise a single dataset, including any supporting or metadata files, it must be enclosed in a single, flat .zip file prior to import. This zip file is what you will upload. Alternatively, if you have a single .csv file from a spreadsheet or a GeoTIFF raster image and do not have a metadata XML file with it, you can upload those files by themselves without zipping them first.

For any dataset, you may include an FGDC XML metadata file, which will be parsed to pre-fill many metadata fields. You should include only one file in the .zip that has the ".shp.xml" extension, and it will be assumed to be an FGDC compliant file. If the .xml is not FGDC compliant, it will be ignored. If you have a structured, indented-text version of a metadata file, first convert it to XML and append a ".shp.xml" extension to it.

All text content, such as .csv and .dbf files, must use UTF-8 character encoding.

Spreadsheets

Delimited text files (.csv) may be imported with the following limitations:

• Files are limited to a maximum of 10,000 records.
• The first row must be a header row with the name of the attribute.
• All rows must have the same number of columns.

You can also create datasets by typing or pasting comma separated values. The same format and size restrictions apply, but this is an easy way to quickly create a new dataset. You can even enter just the headers to define your fields and then make your dataset editable to enter values later. If you don't have latitude/longitude coordinates or a shape file to link to, it is possible to add geocoding information using address, state or zip code information.

Shape files

ESRI Shapefiles are actually several files used together, and may be imported with the following limitations:

• You must include both the .shp and .dbf files, and therefore must zip these together prior to upload.
• A single dataset is limited to a maximum of 10,000 records in the shape file.
• If the shape file is saved with geographic coordinates with respect to the WGS84 geoid. Most GIS (geographic information systems) tools can output shape files in this format. If your shapefile is in a different projection, be sure to also include the .shx and .prj files in your zipped submission.
• Longitude coordinates must be within -180/+180, and may not span the international date line.

All downloaded shape files will be put into zip files, including the .shp, .dbf, .xml, and .idx files. Please note that all maps displayed within Community Insight are in the Mercator projection, and all downloaded shape files are in WGS84.

KML files

KML text files may be imported with the following limitations:

• Only placemarks in the actual document will be imported. KML files sometimes have many embedded links to other files or pieces of content. These are not included in the imported data into Insight.
• Linked content is not included in the imported data.
• All ExtendedData elements in the KML document will be treated as dataset fields.

Since KML is designed for display (like HTML), there can be many variations in the structure of a KML file. It is sometimes necessary to clean up files to ensure all placemarks are in the proper location.

GPX files

GPX text files may be imported with the following limitations:

• Waypoints, tracks and routes may be imported using either GPX 1.0 or 1.1

If a file contains both waypoints and tracks or routes together, they will be split across multiple Insight datasets -- one for the points and one for the paths so both types can be styled appropriately.

GeoTIFF raster imagery

GeoTIFFs are bitmap images which are georeferenced for accurate placement on a map, and may be imported with the following limitations:

• Image is either 3 band or palette-based
• Image must be saved with geographic coordinates with respect to the WGS84 geoid.
• Longitude coordinates must be within -180/+180, and may not span the international date line in a single GeoTIFF.
• Images are limited to a maximum of 10,000 x 10,000 pixels.

Metadata

If you have FGDC metadata in XML format, include it in a .zip with your spreadsheet, ESRI shape files, or GeoTIFF raster imagery. Metadata XML files must include select required FGDC data, including idinfo, distinfo, metainfo elements under the top metadata elements. If these aren't there, the metadata file will be ignored.

Processing

Timing

Once you click Publish, your data will enter a queue to be imported. Community Insight will send you an email when your data is ready. Depending on the size of your dataset and the traffic on our server, this could take between a few minutes and many hours. If you received an email and still can't see your data, go to your library and filter to see just datasets. All of your imported data sets will be there. This collection can be sorted by most recent to put the newest one at the top.