Skip to content

Instantly share code, notes, and snippets.

@tappoz
Created November 12, 2019 19:39
Show Gist options
  • Select an option

  • Save tappoz/2072f6e776ff07914926d8fa472f0746 to your computer and use it in GitHub Desktop.

Select an option

Save tappoz/2072f6e776ff07914926d8fa472f0746 to your computer and use it in GitHub Desktop.

How to normalize nested lists in JSON to a Pandas dataframe

import json
import pandas as pd


my_json_struct = '''
{
	"my_id": 1,
	"my_code": "abc-123",
	"target_date": "2019-11-12",
	"metrics": [{
		"metric_name": "foo-metric",
		"shape": [{
			"start_id": 1,
			"value": 123
		}, {
			"start_id": 48,
			"value": 456
		}]
	}, {
		"metric_name": "bar-metric",
		"shape": [{
			"start_id": 1,
			"value": 987
		}, {
			"start_id": 48,
			"value": 654
		}]
	}]
}
'''

my_dict = json.loads(my_json_struct)

print()
print(my_dict)
print()
print()

my_df = pd.io.json.json_normalize(
            data=my_dict,
            record_path= ['metrics', 'shape'],
            meta= ['my_id', 'my_code', 'target_date', ['metrics', 'metric_name']],
            record_prefix= 'my_struct_',
        )
my_df=my_df[['my_id', 'my_code', 'target_date', 'metrics.metric_name', 'my_struct_start_id', 'my_struct_value']]

print(my_df.head())

Output:

{'my_id': 1, 'my_code': 'abc-123', 'target_date': '2019-11-12', 'metrics': [{'metric_name': 'foo-metric', 'shape': [{'start_id': 1, 'value': 123}, {'start_id': 48, 'value': 456}]}, {'metric_name': 'bar-metric', 'shape': [{'start_id': 1, 'value': 987}, {'start_id': 48, 'value': 654}]}]}


  my_id  my_code target_date metrics.metric_name  my_struct_start_id  my_struct_value
0     1  abc-123  2019-11-12          foo-metric                   1              123
1     1  abc-123  2019-11-12          foo-metric                  48              456
2     1  abc-123  2019-11-12          bar-metric                   1              987
3     1  abc-123  2019-11-12          bar-metric                  48              654
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment