The main specifications for this interface are:
- Use standard statistics
- Allow for multiple subfolders
- Support narrow and wide formats
- Data separated by commas or tabs
- Data is in chronological order by date and time
- Date and time can be within the same column or in separate columns, support both
- Process all files meeting user file name mask (*.csv, *.txt, *.*, and whatever the user may specify)
Testing was conducted to test each detailed specification listed below.
1. In configuration, user selects folder where files/subfolders are located and mask for filename. default mask is '*.CSV', if blank then '*' (one astrick), dropdown options are *.CSV, *.TXT, and *.* - but user can change to anything they wish.
2. Limitations: we cannot process thousand files spanning 10 years of source data. Tested on 30 MB file and on 18 files of more than 1 MB each.
3. Allow user to delete old files and ask for number of days old - if < 7 days then set to 7 days
4. Handle duplicate data points. Compress data to 1 min. even intervals to facilitate filtering on another tag, if multiple values for 1 min, then the last value will be kept - whether wide or narrow format.
5. Handle sub-folders; signal tag will be "FOLDER@@TAG". If using a Filter tag, data for the Filter tag must exist in same folder as Signal tag. If "TAG" is numeric, it denotes a column number, and can only be used when source data files are in Wide format.
6. Ask for date range, if the the requested date range goes past the source date and time - do not process data, but process data upto as close to the end time of the requested range as possible.