How do I parse multiple files in Python and extract important information?

Question

I am trying to write a program that reads data from hundreds of YAML files repetitively and stores certain information contained in the files in a table of some sort. The program would essentially parse every YAML file in a given directory and extract the relevant information until every file has been successfully parsed. An example of the contents of one of the YAML files: %YAML:1.0 camera_rotation_wrt_base: !!opencv-matrix cols: 3 data: [-0.0159428846, 0.0246045925, 0.999570131, -0.999774337, -0.0144301597, -0.0155909406, 0.0140403481, -0.999593139, 0.024829099] dt: f rows: 3 camera_translation_wrt_base: [0.4445618987083435, 0.11700689047574997, 1.5018157958984375] object_rotation_wrt_base: !!opencv-matrix cols: 3 data: [-0.74130547, -0.0615471229, 0.668339849, 0.669196069, -0.144052029, 0.728989482, 0.0514085107, 0.987654269, 0.147973642] dt: f rows: 3 object_rotation_wrt_camera: !!opencv-matrix cols: 3 data: [-0.6565323818253673, 0.1588616842697038, -0.737379262582055, -0.07928983462501557, -0.9866892288165471, -0.14197683014545748, -0.7501189077886223, -0.03474571781315458, 0.6603895951165363] dt: f rows: 3 object_translation_wrt_base: [1.1534364223480225, 0.05951927974820137, 1.3502429723739624] object_translation_wrt_camera: [0.04407151401699165, 0.16979082390232392, 0.705698973194305] template_id: 1965 I would like to be able to store the data key from the object_rotation_wrt_camera key, as well as the object_translation_wrt_camera key, in a CSV file, like so: observation,rotation,translation 1,[-0.53434, 0.023343, .....],[0.54545,0.34344,....] 2,[-0.52234, 0.3433, .....],[0.65645,0.8787344,....] 3,[0.32234, 0.6453, .....],[0.622645,0.1787344,....] In the above table, the observation number pertains to the yaml file, and therefore for each file there is an observation stored in the CSV file for both the rotation and translation variable. (note: the periods used in the table just indicate that the rotation and translation variables continue as they are quite long). Lastly, I would like to create a final CSV file, that is similar to the one above however instead has all of the rotation and translation values separated (meaning that instead of one column for translation and one for rotation, there are 3 for translation pertaining to the 3 values within the lists of the previous CSV file, and 9 columns pertaining to each of the 9 values within the lists of the previous CSV file), like so: observation,tran1,tran2,tran3,rot1,rot2,rot3,rot4,rot5,rot6,rot7,rot8,rot9 1,-0.545434,4.54545,0.343434,............................................. 2,-0.4543,3.3434,0.3534,.................................................. JavaScript questions and answers, JavaScript questions pdf, JavaScript question bank, JavaScript questions and answers pdf, mcq on JavaScript pdf, JavaScript questions and solutions, JavaScript mcq Test , Interview JavaScript questions, JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)

Answer 1

I stored your example in two differently files with .yaml endings and then ran: import sys from pathlib import Path import csv import ruamel.yaml result = [['observation', 'rotation', 'translation']] flatres = ["observation,tran1,tran2,tran3,rot1,rot2,rot3,rot4,rot5,rot6,rot7,rot8,rot9".split(',')] yaml = ruamel.yaml.YAML() for idx, file_name in enumerate(Path('.').glob('*.yaml')): txt = file_name.read_text() if txt.startswith('%YAML:1.0'): txt = txt.replace('%YAML:1.0', "", 1).lstrip() data = yaml.load(txt) result.append([ idx+1, data['object_rotation_wrt_camera']['data'], data['object_translation_wrt_camera'], ]) row = [idx+1] row.extend(data['object_translation_wrt_camera']) row.extend(data['object_rotation_wrt_camera']['data']) flatres.append(row) writer = csv.writer(sys.stdout) writer.writerows(result) print('---------') writer = csv.writer(sys.stdout) writer.writerows(flatres) which gives: observation,rotation,translation 1,"[-0.6565323818253673, 0.1588616842697038, -0.737379262582055, -0.07928983462501557, -0.9866892288165471, -0.14197683014545748, -0.7501189077886223, -0.03474571781315458, 0.6603895951165363]","[0.04407151401699165, 0.16979082390232392, 0.705698973194305]" 2,"[-0.6565323818253673, 0.1588616842697038, -0.737379262582055, -0.07928983462501557, -0.9866892288165471, -0.14197683014545748, -0.7501189077886223, -0.03474571781315458, 0.6603895951165363]","[0.04407151401699165, 0.16979082390232392, 0.705698973194305]" --------- observation,tran1,tran2,tran3,rot1,rot2,rot3,rot4,rot5,rot6,rot7,rot8,rot9 1,0.04407151401699165,0.16979082390232392,0.705698973194305,-0.6565323818253673,0.1588616842697038,-0.737379262582055,-0.07928983462501557,-0.9866892288165471,-0.14197683014545748,-0.7501189077886223,-0.03474571781315458,0.6603895951165363 2,0.04407151401699165,0.16979082390232392,0.705698973194305,-0.6565323818253673,0.1588616842697038,-0.737379262582055,-0.07928983462501557,-0.9866892288165471,-0.14197683014545748,-0.7501189077886223,-0.03474571781315458,0.6603895951165363 Because there are commas in the sequences that you want to have stored, those entries need to be quoted and Python's CSV writer does that automatically (otherwise the second and following rows of the CSV would have many more than three elements). YAML 1.0 was superseded, by YAML 1.1, in 2005 and I have not paid special attention to the differences between those versions apart from the directive. YAML 1.2 has been the YAML standard since 2009. ruamel.yaml only supports files that have explicit YAML 1.2 (or 1.1) directives, that is why the %YAML:1.0 directive needs to be explicitly removed from these files. You might run into trouble if you have any "old-style" octal integers in your files and in some other cases not shown in your input example.