Java Flat File Processor

In the 90s, when I’ve started seen computers and similar things, we use to go to the files and check them using a text editor. In most of the cases such files were unreadable binary files, but from time to time it was usual to find text-based files with readable content in them.

source: https://github.com/modusbox/datasonnet-flat-file-examples/blob/master/src/test/resources/mapping-tests/flat-file-to-xml.ds/example/inputs/payload.txt

order12345,"04/11/2020 14:35:00"
"Kym Chillcot","6904 Vera Trail","White Plains",NY,10633,"(845) 568-0141","kchillcot0@washingtonpost.com"
shipment1,"04/21/2020 09:10:00"
"Ava McGunley","5488 Katie Plaza","Las Vegas",NV,89105,"(702) 594-4721",amcgunley2@forbes.com
"Hand Sanitizer",hs123,2
shipment2,"04/23/2020 08:42:00"
"Ava McGunley","5488 Katie Plaza","Las Vegas",NV,89105,"(702) 594-4721",amcgunley2@forbes.com
"Toilet Paper",tp2pl976,12
"Face Mask",856326,10

If I recall correctly, once we got into a school-system, where everything was written in a big file, for the entire school… we just played around with our grades a bit.

But since then I’ve always felt that flat files had a bit of history, it described how these systems were designed and developed, it showed the discipline (or the lack of it) from the teams or individuals producing such files. These files were meant to be readable, but they had a strange structure that was only understandable if we got to structurize such data.

Back then, we thought that it would be a great idea to write an application to parse such files and identify automatically the data. Sadly, such idea was not simple to implement then, and it’s still something really difficult now.

For years we’ve seen integrations with legacy systems using flat-files, most of the times the legacy-system’s owners provided some sort of library to read such files, but since developers are now going into retirement, these libraries are usually obsolete.

Now we are facing the situation, in which we’ve implemented several flat-file parsers in our own because the existing libraries simply didn’t fit our needs, perhaps we can do something different and simple that can be helpful to someone else dealing with these kind of files.

We’ll see…