#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
The datamodel classes constitute a data model implementation
meant to mirror the functionality of the data model output from pycdf, though
implemented slightly differently.

This contains the following classes:
 * :py:class:`dmarray` - numpy arrays that support .attrs for information about the data
 * :py:class:`SpaceData` - base class that extends dict, to be extended by others

Authors: Steve Morley and Brian Larsen

Additional Contributors: Charles Kiyanda and Miles Engel

Institution: Los Alamos National Laboratory

Contact: smorley@lanl.gov; balarsen@lanl.gov

Copyright 2010-2016 Los Alamos National Security, LLC.


About datamodel
---------------

The SpacePy datamodel module implements classes that are designed to make implementing a standard
data model easy. The concepts are very similar to those used in standards like HDF5, netCDF and
NASA CDF.

The basic container type is analogous to a folder (on a filesystem; HDF5 calls this a
group): Here we implement this as a dictionary-like object, a datamodel.SpaceData object, which
also carries attributes. These attributes can be considered to be global, i.e. relevant for the
entire folder. The next container type is for storing data and is based on a numpy array, this
class is datamodel.dmarray and also carries attributes. The dmarray class is analogous to an
HDF5 dataset.

In fact, HDF5 can be loaded directly into a SpacePy datamodel, carrying across all attributes,
using the function fromHDF5:

>>> import spacepy.datamodel as dm
>>> data = dm.fromHDF5('test.h5')

Functions are also available to directly load data and metadata into a SpacePy datamodel from
NASA CDF as well as JSON-headed ASCII. Writers also exist to output a SpacePy datamodel directly
to HDF5 or JSON-headed ASCII. See :py:func:`datamodel.fromCDF`, :py:func:`datamodel.readJSONheadedASCII`,
:py:func:`datamodel.toHDF5`, and :py:func:`datamodel.toJSONheadedASCII` for more details.


Examples
--------

Imagine representing some satellite data within the global attributes might be
the mission name and the instrument PI, the variables might be the
instrument counts [n-dimensional array], timestamps[1-dimensional array and an orbit number [scalar].
Each variable will have one attribute (for this example).

>>> import spacepy.datamodel as dm
>>> mydata = dm.SpaceData(attrs={'MissionName': 'BigSat1'})
>>> mydata['Counts'] = dm.dmarray([[42, 69, 77], [100, 200, 250]], attrs={'Units': 'cnts/s'})
>>> mydata['Epoch'] = dm.dmarray([1, 2, 3], attrs={'units': 'minutes'})
>>> mydata['OrbitNumber'] = dm.dmarray(16, attrs={'StartsFrom': 1})
>>> mydata.attrs['PI'] 'Prof. Big Shot'

This has now populated a structure that can map directly to a NASA CDF, HDF5 or JSON-headed ASCII file.
To visualize our datamodel, we can use tree method (which can be applied to any dictionary-like object
using :func:`~spacepy.toolbox.dictree`).

>>> mydata.tree(attrs=True)

::

    +
    :|____MissionName
    :|____PI
    |____Counts
         :|____Units
    |____Epoch
         :|____units
    |____OrbitNumber
         :|____StartsFrom


Guide for NASA CDF users
------------------------
By definition, a NASA CDF only has a single 'layer'. That is, a CDF contains a series of records
(stored variables of various types) and a set of attributes that are either global or local in
scope. Thus to use SpacePy's datamodel to capture the functionality of CDF the two basic data types
are all that is required, and the main constraint is that datamodel.SpaceData objects cannot be
nested (more on this later, if conversion from a nested datamodel to a flat datamodel is required).


Opening a CDF and working directly with the contents can be easily done using the PyCDF module, however,
if you wish to load the entire contents of a CDF directly into a datamodel (complete with attributes)
the following will make life easier:

>>> import spacepy.datamodel as dm
>>> data = dm.fromCDF('inFile.cdf')


A quick guide to JSON-headed ASCII
----------------------------------
In many cases it is preferred to have a human-readable ASCII file, rather than a binary file like CDF
or HDF5. To make it easier to carry all the same metadata that is available in HDF5 or CDF we have
developed an ASCII data storage format that encodes the metadata using JSON (JavaScript Object Notation).
This notation supports two basic datatypes: key/value collections (like a SpaceData) and ordered lists
(which can represent arrays). JSON is human-readable, but if large arrays are stored in metadata is quickly
becomes difficult to read. For this reason we use JSON to encode the metadata (usually smaller datasets)
and store the data in a standard flat-ASCII format. The metadata is provided as a header that describes
the contents of the file.


To use JSON for storing only metadata associated with the data to be written to an ASCII file a minimal
metadata standard must be implemented. We use the following attribute names: DIMENSION and START_COLUMN.
We also recommend using the NASA ISTP metadata standard to assign attribute names. The biggest limitation
of flat ASCII is that sensibly formatting datasets of more than 2-dimensions (i.e. ranks greater than 2)
is not possible. For this reason if you have datasets of rank 3 or greater then we recommend using HDF5.
If text is absolutely required then it is possible to encode multi-dimensional arrays in the JSON metadata,
but this is not recommended.


This format is best understood by illustration. The following example builds a toy SpacePy datamodel and
writes it to a JSON-headed ASCII file. The contents of the file are then shown.

>>> import spacepy.datamodel as dm
>>> data = dm.SpaceData()
>>> data.attrs['Global'] = 'A global attribute'
>>> data['Var1'] = dm.dmarray([1,2,3,4,5], attrs={'Local1': 'A local attribute'})
>>> data['Var2'] = dm.dmarray([[8,9],[9,1],[3,4],[8,9],[7,8]])
>>> data['MVar'] = dm.dmarray([7.8], attrs={'Note': 'Metadata'})
>>> dm.toJSONheadedASCII('outFile.txt', data, depend0='Var1', order=['Var1'])
#Note that not all field names are required, those not given will be listed
#alphabetically after those that are specified

The file looks like:

.. code-block:: none

    #{
    #    "MVar": {
    #        "Note": "Metadata",
    #        "VALUES": [7.8]
    #    },
    #    "Global": "A global attribute",
    #    "Var1": {
    #        "Local1": "A local attribute",
    #        "DIMENSION": [1],
    #        "START_COLUMN": 0
    #    },
    #    "Var2": {
    #        "DIMENSION": [2],
    #        "START_COLUMN": 2
    #    }
    #}
    1 8 9
    2 9 1
    3 3 4
    4 8 9
    5 7 8

"""

from __future__ import division
import copy
import datetime
import itertools
import json
from functools import partial
import os
import re
import warnings

try:
    import StringIO # can't use cStringIO as we might have unicode
except ImportError:
    import io as StringIO

import numpy
# from . import toolbox # handled in functions that use it


__contact__ = 'Steve Morley, smorley@lanl.gov'

# python2 python3 string wrangling
try:
    str_classes = (str, bytes, unicode)
except NameError:
    str_classes = (str, bytes)
    unicode = str

class DMWarning(Warning):
    """
    Warnings class for datamodel, subclassed so it can be set to always
    """
    pass
warnings.simplefilter('always', DMWarning)

class MetaMixin(object):
    """Mixin class that adds a 'meta' attribute that acts like 'attrs'

    Recommendation from the Python Heliophysics community is to allow
    access to metadata via either an ``attrs`` attribute or ``meta``.
    This mixin class supports that recommendation.
    """
    
    @property
    def meta(self):
        """Equivalent to ``attrs``

        Some APIs use ``attrs`` for metadata; some use ``meta``. This
        is a convenience property to make it easier for those familiar
        with the ``meta`` convention.
        """
        return self.attrs

    @meta.setter
    def meta(self, v):
        """Set meta as with attrs"""
        self.attrs = v

    @meta.deleter
    def meta(self):
        """Remove meta (and thus attrs)

        This isn't a good idea but you can do it with attrs,
        so might as well support it in meta.

        This still leaves the meta property hanging around, but
        cannot delete a property from an instance (have to delete
        from the class).
        """
        del self.attrs


class dmarray(numpy.ndarray, MetaMixin):
    """
    Container for data within a SpaceData object

    Raises
    ------
    NameError
        raised is the request name was not added to the allowed attributes list

    Examples
    --------
    >>> import spacepy.datamodel as datamodel
    >>> position = datamodel.dmarray([1,2,3], attrs={'coord_system':'GSM'})
    >>> position
    dmarray([1, 2, 3])
    >>> position.attrs
    {'coord_system': 'GSM'}a

    The dmarray, like a numpy ndarray, is versatile and can store
    any datatype; dmarrays are not just for arrays.

    >>> name = datamodel.dmarray('TestName')
    dmarray('TestName')

    To extract the string (or scalar quantity), use the tolist method

    >>> name.tolist()
    'TestName'

    .. currentmodule:: spacepy.datamodel
    .. autosummary::
        ~dmarray.addAttribute
    .. automethod:: addAttribute
    """
    Allowed_Attributes = ['attrs']

    def __new__(cls, input_array, attrs=None, dtype=None):
       # Input array is an already formed ndarray instance
       # We first cast to be our class type
       if not dtype:
           obj = numpy.asarray(input_array).view(cls)
       else:
           obj = numpy.asarray(input_array).view(cls).astype(dtype)
       # add the new attribute to the created instance
       if attrs != None:
           obj.attrs = attrs
       else:
           obj.attrs = {}
       # Finally, return the newly created object:
       return obj

    def __array_finalize__(self, obj):
       # see InfoArray.__array_finalize__ for comments
        if obj is None:
            return
        for val in self.Allowed_Attributes:
            self.__setattr__(val, copy.deepcopy(getattr(obj, val, {})))

    def __array_wrap__(self, out_arr, context=None):
        #check for zero-dims (numpy bug means subclass behaviour isn't consistent with ndarray
        #this traps most of the bad behaviour ( std() and var() still problems)
        if out_arr.ndim > 0:
            return numpy.ndarray.__array_wrap__(self, out_arr, context)
        else:
            return numpy.ndarray.__array_wrap__(self, out_arr, context).tolist()

    def __reduce__(self):
        """This is called when pickling, see:
        http://www.mail-archive.com/numpy-discussion@scipy.org/msg02446.html
        for this particular example.
        Only the attributes in Allowed_Attributes can exist
        """
        object_state = list(numpy.ndarray.__reduce__(self))
        subclass_state = tuple([tuple([val, self.__getattribute__(val)]) for val in self.Allowed_Attributes])
        object_state[2] = (object_state[2],subclass_state)
        return tuple(object_state)

    def __setstate__(self, state):
        """Used for unpickling after __reduce__ the self.attrs is recovered from
        the way it was saved and reset.
        """
        nd_state, own_state = state
        numpy.ndarray.__setstate__(self,nd_state)
        for i, val in enumerate(own_state):
            if not val[0] in self.Allowed_Attributes: # this is attrs
                self.Allowed_Attributes.append(own_state[i][0])
            self.__setattr__(own_state[i][0], own_state[i][1])

    def __setattr__(self, name, value):
        """Make sure that .attrs is the only attribute that we are allowing
        dmarray_ne took 15.324803 s
        dmarray_eq took 15.665865 s
        dmarray_assert took 16.025478 s
        It looks like != is the fastest, but not by much over 10000000 __setattr__
        """
        #meta is special-handled because it should NOT be pickled
        if name in ('Allowed_Attributes', 'meta'):
            pass
        elif not name in self.Allowed_Attributes:
            raise(TypeError("Only attribute listed in Allowed_Attributes can be set"))
        super(dmarray, self).__setattr__(name, value)

    def addAttribute(self, name, value=None):
        """Method to add an attribute to a dmarray
        equivalent to
        a = datamodel.dmarray([1,2,3])
        a.Allowed_Attributes = a.Allowed_Attributes + ['blabla']
        """
        if name in self.Allowed_Attributes:
            raise(NameError('{0} is already an attribute cannot add again'.format(name)))
        self.Allowed_Attributes.append(name)
        self.__setattr__(name, value)

    def count(self, srchval):
        """
        Equivalent to count method on list

        """
        mask = self == srchval
        return int(mask.sum())
    def _saveAttrs(self):
        Allowed_Attributes = self.Allowed_Attributes
        backup = []
        for atr in Allowed_Attributes:
            backup.append( (atr, dmcopy(self.__getattribute__(atr)) ) )
        return backup

    @classmethod
    def _replaceAttrs(cls, arr, backup):
        for key, val in backup:
            if key != 'attrs':
                try:
                    arr.addAttribute(key)
                except NameError:
                    pass
            arr.__setattr__(key, val)
        return arr

    @classmethod
    def append(cls, one, other):
        """
        append data to an existing dmarray
        """
        backup = one._saveAttrs()
        outarr = dmarray(numpy.append(one, other))
        return cls._replaceAttrs(outarr, backup)

    @classmethod
    def vstack(cls, one, other):
        """
        vstack data to an existing dmarray
        """
        backup = one._saveAttrs()
        outarr = dmarray(numpy.vstack( (one, other) ))
        return cls._replaceAttrs(outarr, backup)

    @classmethod
    def hstack(cls, one, other):
        """
        hstack data to an existing dmarray
        """
        backup = one._saveAttrs()
        outarr = dmarray(numpy.hstack( (one, other) ))
        return cls._replaceAttrs(outarr, backup)

    @classmethod
    def dstack(cls, one, other):
        """
        dstack data to an existing dmarray
        """
        backup = one._saveAttrs()
        outarr = dmarray(numpy.dstack( (one, other) ))
        return cls._replaceAttrs(outarr, backup)

    @classmethod
    def concatenate(cls, one, other, axis=0):
        """
        concatenate data to an existing dmarray
        """
        backup = one._saveAttrs()
        outarr = dmarray(numpy.concatenate( (one, other) , axis=axis ))
        return cls._replaceAttrs(outarr, backup)

def dmfilled(shape, fillval=0, dtype=None, order='C', attrs=None):
    """
    Return a new dmarray of given shape and type, filled with a specified value (default=0).

    See Also
    --------
    numpy.ones

    Examples
    --------
    >>> import spacepy.datamodel as dm
    >>> dm.dmfilled(5, attrs={'units': 'nT'})
    dmarray([ 0.,  0.,  0.,  0.,  0.])

    >>> dm.dmfilled((5,), fillval=1, dtype=np.int)
    dmarray([1, 1, 1, 1, 1])

    >>> dm.dmfilled((2, 1), fillval=np.nan)
    dmarray([[ nan],
           [ nan]])

    >>> a = dm.dmfilled((2, 1), np.nan, attrs={'units': 'nT'})
    >>> a
    dmarray([[ nan],
           [ nan]])
    >>> a.attrs
    {'units': 'nT'}
        
    """
    a = dmarray(numpy.empty(shape, dtype, order), attrs=attrs)
    a.fill(fillval)
    return a


class SpaceData(dict, MetaMixin):
    """
    Datamodel class extending dict by adding attributes.

    .. currentmodule:: spacepy.datamodel
    .. autosummary::
        ~SpaceData.flatten
        ~SpaceData.tree
        ~SpaceData.toCDF
        ~SpaceData.toHDF5
        ~SpaceData.toJSONheadedASCII
    .. automethod:: flatten
    .. automethod:: tree
    .. automethod:: toCDF
    .. automethod:: toHDF5
    .. automethod:: toJSONheadedASCII
    """
    def __getitem__(self, key):
        """
        This allows one to make a SpaceData indexed with an iterable of keys to return a new spacedata
        made of the subset of keys
        """
        try: 
            return super(SpaceData, self).__getitem__(key)
        except (KeyError, TypeError):
            if isinstance(key, (tuple, list)):
                # make a new SpaceData from these keys
                out = SpaceData()
                out.attrs = self.attrs
                for k in key:
                    out[k] = self[k]
                return out
            else:
                raise(KeyError('{0}'.format(key)))

    def __init__(self, *args, **kwargs):
        """
        Base class for "Data Model" representation data

        Abstract method, reimplement

        Attributes
        ----------
        attrs : dict
            dictionary of the attributes of the SpaceData object

        """
        #raise(ValueError("Abstract method called, reimplement __init__"))
        self.attrs = {}
        if 'attrs' in kwargs:
            if hasattr(kwargs['attrs'], '__getitem__'):
                self.attrs = kwargs['attrs']
            del kwargs['attrs']

        super(SpaceData, self).__init__(*args, **kwargs)
        self.toCDF = partial(toCDF, SDobject=self)
        self.toCDF.__doc__ = toCDF.__doc__
        self.toHDF5 = partial(toHDF5, SDobject=self)
        self.toHDF5.__doc__ = toHDF5.__doc__
        self.toJSONheadedASCII = partial(toJSONheadedASCII, insd=self)
        self.toJSONheadedASCII.__doc__ = toJSONheadedASCII.__doc__

## To enable string output of repr, instead of just printing, uncomment his block
#    def __repr__(self):
#        #redirect stdout to StringIO
#        import StringIO, sys
#        dum = StringIO.StringIO()
#        sys_stdout_save = sys.stdout
#        sys.stdout = dum
#        self.tree(verbose=True)
#        sys.stdout = sys_stdout_save
#        dum.seek(0)
#        return ''.join(dum.readlines())
    
    def tree(self, **kwargs):
        '''Print the contents of the SpaceData object in a visual tree

        Other Parameters
        ----------------
        verbose : boolean (optional)
            print more info
        spaces : string (optional)
            string will added for every line
        levels : integer (optional)
            number of levels to recurse through (True means all)
        attrs : boolean (optional)
            display information for attributes

        Examples
        --------
        >>> import spacepy.datamodel as dm
        >>> import spacepy.toolbox as tb
        >>> a = dm.SpaceData()
        >>> a['1'] = dm.SpaceData(dog = 5)
        >>> a['4'] = dm.SpaceData(cat = 'kitty')
        >>> a['5'] = 4
        >>> a.tree()
        +
        |____1
             |____dog
        |____4
             |____cat
        |____5

        See Also
        --------
        toolbox.dictree
        '''
        from . import toolbox
        toolbox.dictree(self, **kwargs)

    def flatten(self):
        '''
        Method to collapse datamodel to one level deep

        Examples
        --------
        >>> import spacepy.datamodel as dm
        >>> import spacepy.toolbox as tb
        >>> a = dm.SpaceData()
        >>> a['1'] = dm.SpaceData(dog = 5, pig = dm.SpaceData(fish=dm.SpaceData(a='carp', b='perch')))
        >>> a['4'] = dm.SpaceData(cat = 'kitty')
        >>> a['5'] = 4
        >>> a.tree()
        +
        |____1
             |____dog
             |____pig
                  |____fish
                       |____a
                       |____b
        |____4
             |____cat
        |____5

        >>> b = dm.flatten(a)
        >>> b.tree()
        +
        |____1<--dog
        |____1<--pig<--fish<--a
        |____1<--pig<--fish<--b
        |____4<--cat
        |____5

        >>> a.flatten()
        >>> a.tree()
        +
        |____1<--dog
        |____1<--pig<--fish<--a
        |____1<--pig<--fish<--b
        |____4<--cat
        |____5

        '''
        flatobj = flatten(self)
        remkeys = [key for key in self]
        for key in remkeys:
            del self[key]
        for key in flatobj:
            self[key] = copy.copy(flatobj[key])

            

def convertKeysToStr(SDobject):
    if isinstance(SDobject, SpaceData):
        newSDobject = SpaceData()
        newSDobject.attrs = SDobject.attrs
    else:
        newSDobject = {}
    for key in SDobject:
        if not isinstance(key, str_classes):
            if isinstance(SDobject[key], dict):
                newSDobject[str(key)] = convertKeysToStr(SDobject[key])
            else:
                newSDobject[str(key)] = SDobject[key]
        else:
            if isinstance(SDobject[key], dict):
                newSDobject[key] = convertKeysToStr(SDobject[key])
            else:
                newSDobject[key] = SDobject[key]

    return newSDobject


def flatten(dobj):
    '''Collapse datamodel to one level deep

    Examples
    --------

    >>> import spacepy.datamodel as dm
    >>> import spacepy.toolbox as tb
    >>> a = dm.SpaceData()
    >>> a['1'] = dm.SpaceData(dog = 5, pig = dm.SpaceData(fish=dm.SpaceData(a='carp', b='perch')))
    >>> a['4'] = dm.SpaceData(cat = 'kitty')
    >>> a['5'] = 4
    >>> a.tree()
    +
    |____1
         |____dog
         |____pig
              |____fish
                   |____a
                   |____b
    |____4
         |____cat
    |____5

    >>> b = dm.flatten(a)
    >>> b.tree()
    +
    |____1<--dog
    |____1<--pig<--fish<--a
    |____1<--pig<--fish<--b
    |____4<--cat
    |____5

    >>> a.flatten()
    >>> a.tree()
    +
    |____1<--dog
    |____1<--pig<--fish<--a
    |____1<--pig<--fish<--b
    |____4<--cat
    |____5


    See Also
    --------
    unflatten
    SpaceData.flatten

    '''

    try:
        addme = dobj.__class__()
    except (TypeError):
        addme = SpaceData()
    remlist = []
    for key in dobj: #iterate over keys in SpaceData
        if isinstance(dobj[key], dict):
            remlist.append(key)
            newname = str(key) + '<--'
            for levkey in dobj[key]:
                if hasattr(dobj[key][levkey], 'keys'):
                    retdict = flatten(dobj[key][levkey])
                    for key2 in retdict:
                        addme[newname+levkey+'<--'+key2] = retdict[key2]
                else:
                    addme[newname+levkey] = copy.copy(dobj[key][levkey])
        else:
            addme[key] = copy.copy(dobj[key])
    return addme

def unflatten(dobj, marker='<--'):
    '''Collapse datamodel to one level deep

    Examples
    --------

    >>> import spacepy.datamodel as dm
    >>> import spacepy.toolbox as tb
    >>> a = dm.SpaceData()
    >>> a['1'] = dm.SpaceData(dog = 5, pig = dm.SpaceData(fish=dm.SpaceData(a='carp', b='perch')))
    >>> a['4'] = dm.SpaceData(cat = 'kitty')
    >>> a['5'] = 4
    >>> a.tree()
    +
    |____1
         |____dog
         |____pig
              |____fish
                   |____a
                   |____b
    |____4
         |____cat
    |____5

    >>> b = dm.flatten(a)
    >>> b.tree()
    +
    |____1<--dog
    |____1<--pig<--fish<--a
    |____1<--pig<--fish<--b
    |____4<--cat
    |____5

    >>> c = dm.unflatten(b)
    >>> c.tree()
    +
    |____1
         |____dog
         |____pig
              |____fish
                   |____a
                   |____b
    |____4
         |____cat
    |____5


    '''
    #set up a new object for return
    try:
        addme = dobj.__class__()
    except (TypeError):
        addme = SpaceData()
    #the input is assumed to be single level (i.e. it is flat)

    #find all keys that have at least one marker,
    #then unpack. Recurse over these until no more markers are found.
    keydict = {}
    for key in dobj:
        if isinstance(dobj[key], dict):
            raise TypeError('Flat datamodel should not contain dict-likes')
        try:
            if marker in key:
                #get 'group'
                group = key.split(marker)[0]
                if not group in keydict:
                    keydict[group] = {key: ''}
                else:
                    keydict[group][key] = ''
            else: #not nested, just copy key
                addme[key] = dmcopy(dobj[key])
        except:
            addme[key] = dmcopy(dobj[key])
    #now we have all the groups at this level
    #move members of groups into new SpaceDatas
    for grp in keydict:
        addme[grp] = SpaceData()
        for key in keydict[grp]:
            newkey = marker.join(key.split(marker)[1:])
            addme[grp][newkey] = dmcopy(dobj[key])
        addme[grp] = unflatten(addme[grp], marker=marker) #recurse to make sure everything inside is unpacked
    return addme


def fromCDF(fname, **kwargs):
    '''
    Create a SpacePy datamodel representation of a NASA CDF file

    Parameters
    ----------
    file : string
        the name of the cdf file to be loaded into a datamodel

    Returns
    -------
    out : spacepy.datamodel.SpaceData
        SpaceData with associated attributes and variables in dmarrays

    Examples
    --------
    >>> import spacepy.datamodel as dm
    >>> data = dm.fromCDF('test.cdf')

    See Also
    --------
    spacepy.pycdf.CDF.copy
    '''
    #TODO: add unflatten keyword and restore flattened variables
    try:
        from spacepy import pycdf
    except ImportError:
        raise ImportError("CDF converter requires NASA CDF library and SpacePy's pyCDF")

    with pycdf.CDF(fname) as cdfdata:
        return cdfdata.copy()

def toCDF(fname, SDobject, **kwargs):
    '''
    Create a CDF file from a SpacePy datamodel representation

    Parameters
    ----------
    fname : str
        Filename to write to

    SDobject : spacepy.datamodel.SpaceData
        SpaceData with associated attributes and variables in dmarrays

    Other Parameters
    ----------------
    skeleton : str (optional)
        create new CDF from a skeleton file (default '')

    flatten : bool (optional)
        flatten incoming datamodel - if SpaceData objects are nested (default False)

    overwrite : bool (optional)
        allow overwrite of an existing target file (default False)

    autoNRV : bool (optional)
        attempt automatic identification of non-record varying entries in CDF

    backward : bool (optional)
        create CDF in backward-compatible format (default is v3+ compatibility only)

    TT2000 : bool (optional)
        write variables beginning with 'Epoch' as datatype CDF_TT2000 (default is automatic selection of EPOCH or EPOCH16)

    verbose : bool (optional)
        verbosity flag

    Returns
    -------
    None
    '''
    defaults = {'skeleton': '',
                'flatten': False,
                'overwrite': False,
                'compress': False,
                'autoNRV': False,
                'backward': False,
                'TT2000': False,
                'verbose': False}
    for key in kwargs:
        if key in defaults:
            defaults[key] = kwargs[key]

    if defaults['flatten']:
        SDobject = SDobject.flatten()
    if defaults['overwrite']:
        raise NotImplementedError('Overwriting CDFs is not currently enabled - please remove the file manually')

    try:
        from spacepy import pycdf
    except ImportError:
        raise ImportError("CDF converter requires NASA CDF library and SpacePy's pyCDF")
    pycdf.lib.set_backward(False)
    with pycdf.CDF(fname, defaults['skeleton']) as outdata:
        if hasattr(SDobject, 'attrs'):
            for akey in SDobject.attrs:
                outdata.attrs[akey] = dmcopy(SDobject.attrs[akey])
        varLengths = [len(SDobject[var]) for var in SDobject]
        modeLength = next(itertools.groupby((reversed(sorted(varLengths)))))[0]
        for key in SDobject:
            if isinstance(SDobject[key], dict):
                raise TypeError('This data structure appears to be nested, please try spacepy.datamodel.flatten')
            if not defaults['skeleton']:
                if not SDobject[key].shape:
                    shape_tup=-1
                else:
                    shape_tup = SDobject[key].shape
                if 'Epoch' not in SDobject:
                    NRVtest = modeLength
                else:
                    NRVtest = len(SDobject['Epoch'])
                if shape_tup[0] != NRVtest: #naive check for 'should-be' NRV
                    try:
                        foo = outdata.new(key, SDobject[key][...], recVary=False)
                        if defaults['verbose']: print('{0} is being made NRV'.format(key))
                        outdata[key].attrs = dmcopy(SDobject[key].attrs)
                    except ValueError:
                        foo = outdata.new(key, SDobject[key].tolist, recVary=False)
                        outdata[key].attrs = dmcopy(SDobject[key].attrs)
                if defaults['TT2000'] and 'Epoch' in key:
                    foo = outdata.new(key, SDobject[key][...], type=pycdf.const.CDF_TIME_TT2000)
                else:
                    try:
                        outdata[key] = SDobject[key]
                    except ValueError:
                        try:
                            outdata[key] = dmarray([SDobject[key].tolist()], attrs=dmcopy(SDobject[key].attrs)).squeeze()
                        except UnicodeEncodeError:
                            tmpAttrs = dmcopy(SDobject[key].attrs)
                            for akey in tmpAttrs:
                                try: #strings
                                    if hasattr(tmpAttrs[akey], 'encode'):
                                        tmpAttrs[akey] = tmpAttrs[akey].encode('utf-8')
                                    else:
                                        tmpAttrs[akey] = tmpAttrs[akey]
                                except AttributeError: #probably a list of strings
                                    for id, el in enumerate(tmpAttrs[akey]):
                                        tmpAttrs[akey][id] = el.encode('utf-8')

            else:
                outdata[key][...] = SDobject[key][...]
                for akey in outdata[key].attrs:
                    try:
                        outdata[key].attrs[akey] = dmcopy(SDobject[key].attrs[akey])
                    except ValueError:
                        outdata[key][...] = dmarray([SDobject[key].tolist()], attrs=dmcopy(SDobject[key].attrs))
                    except KeyError:
                        pass
    return None


def fromHDF5(fname, **kwargs):
    '''
    Create a SpacePy datamodel representation of an HDF5 file or netCDF4 file which is HDF5 compliant

    Parameters
    ----------
    file : string
        the name of the HDF5/netCDF4 file to be loaded into a datamodel

    Returns
    -------
    out : spacepy.datamodel.SpaceData
        SpaceData with associated attributes and variables in dmarrays

    Examples
    --------
    >>> import spacepy.datamodel as dm
    >>> data = dm.fromHDF5('test.hdf')

    Notes
    -----
    Zero-sized datasets will break in h5py. This is kluged by returning a
    dmarray containing a None.

    This function is expected to work with any HDF5-compliant files, including
    netCDF4 (not netCDF3) and MatLab save files from v7.3 or later, but some
    datatypes are not supported, e.g., non-string vlen datatypes, and will
    raise a warning.
    '''
    def hdfcarryattrs(SDobject, hfile, path):
        if hasattr(hfile[path],'attrs'):
            #for key, value in hfile[path].attrs.iteritems():
            for key in hfile[path].attrs:
                try:
                    value = hfile[path].attrs[key]
                except TypeError:
                    warnings.warn('Unsupported datatype in dataset {0}.attrs[{1}]'.format(path,key))
                    continue
                try:
                    SDobject.attrs[key] = value
                except:
                    warnings.warn('The following key:value pair is not permitted\n' +
                                    'key = {0} ({1})\n'.format(key, type(key)) +
                                    'value = {0} ({1})'.format(value, type(value)), DMWarning)

    try:
        import h5py as hdf
    except ImportError:
        raise ImportError('HDF5 converter requires h5py')

    if type(fname) in str_classes:
        hfile = hdf.File(fname, mode='r')
    else:
        hfile = fname
        #should test here for HDF file object

    if 'path' not in kwargs:
        path = '/'
    else:
        path = kwargs['path']

    SDobject = SpaceData()
    allowed_elems = [hdf.Group, hdf.Dataset]
    ##carry over the attributes
    hdfcarryattrs(SDobject, hfile, path)
    ##carry over the groups and datasets
    for key, value in hfile[path].items():
        #try:
            if type(value) is allowed_elems[0]: #if a group
                SDobject[key] = SpaceData()
                SDobject[key] = fromHDF5(hfile, path=path+'/'+key)
            elif type(value) is allowed_elems[1]: #if a dataset
                try:
                    SDobject[key] = dmarray(value)
                except (TypeError, ZeroDivisionError): #ZeroDivisionError catches zero-sized DataSets
                    SDobject[key] = dmarray(None)
                hdfcarryattrs(SDobject[key], hfile, path+'/'+key)
        #except:
        #    raise ValueError('HDF5 file contains type other than Group or Dataset')
    if path=='/': hfile.close()
    return SDobject

def toHDF5(fname, SDobject, **kwargs):
    '''
    Create an HDF5 file from a SpacePy datamodel representation

    Parameters
    ----------
    fname : str
        Filename to write to

    SDobject : spacepy.datamodel.SpaceData
        SpaceData with associated attributes and variables in dmarrays

    Other Parameters
    ----------------
    overwrite : bool (optional)
        allow overwrite of an existing target file (default True)
    mode : str (optional)
        HDF5 file open mode (a, w, r) (default 'a')
    compression : str (optional)
        compress all the variables using this method (default None) (gzip, shuffle, fletcher32, szip, lzf)
    compression_opts : str (optional)
        options to the compression, see h5py documentation for more details

    Returns
    -------
    None

    Examples
    --------
    >>> import spacepy.datamodel as dm
    >>> a = dm.SpaceData()
    >>> a['data'] = dm.dmarray(range(100000), dtype=float)
    >>> dm.toHDF5('test_gzip.h5', a, overwrite=True, compression='gzip')
    >>> dm.toHDF5('test.h5', a, overwrite=True)
    >>> # test_gzip.h5 was 118k, test.h5 was 785k
    '''
    def SDcarryattrs(SDobject, hfile, path, allowed_attrs):
        if hasattr(SDobject, 'attrs'):
            for key, value in SDobject.attrs.items():
                dumval, dumkey = copy.copy(value), copy.copy(key)
                if type(value) in allowed_attrs:
                    #test for datetimes in iterables
                    if hasattr(value, '__iter__') and not isinstance(value, str_classes):
                        dumval = [b.isoformat() if isinstance(b, datetime.datetime) else b for b in value]
                    truth = False
                    try:
                        if value.nbytes: truth = True #empty arrays of any dimension are nbytes=0
                    except AttributeError: #not an array
                        if value or value == 0: truth = True

                    if truth:
                        if bytes is str:
                            if type(key) is unicode:
                                dumkey = key.encode('utf-8')
                            if type(value) is unicode:
                                dumval = value.encode('utf-8')
                        uni = False #No special unicode handling
                        if not bytes is str: #Python 3
                            dumval = numpy.asanyarray(dumval)
                            if dumval.size and dumval.dtype.kind == 'U':
                                uni = True #Unicode list, special handling
                        try:
                            if uni:
                                #Tell hdf5 this is unicode. Numpy is UCS-4, HDF5 is UTF-8
                                hfile[path].attrs.create(dumkey, dumval,
                                    dtype=hdf.special_dtype(vlen=unicode))
                            else:
                                hfile[path].attrs[dumkey] = dumval
                        except TypeError:
                            hfile[path].attrs[dumkey] = str(dumval)
                            warnings.warn(
                                'The following value is not permitted\n' +
                                'key, value, type = {0}, {1}, {2})\n'.format(
                                    key, value, type(value)) +
                                'value has been converted to a string for output',
                                DMWarning)
                    else:
                        hfile[path].attrs[dumkey] = ''
                elif isinstance(value, datetime.datetime):
                    dumval = value.isoformat()
                    if bytes is str and type(key) is unicode:
                        dumkey = str(key)
                    hfile[path].attrs[dumkey] = dumval
                else:
                    #TODO: add support for arrays(?) in attrs (convert to isoformat)
                    warnings.warn('The following key:value pair is not permitted\n' +
                                    'key = {0} ({1})\n'.format(key, type(key)) +
                                    'value type {0} is not in the allowed attribute list'.format(type(value)),
                                        DMWarning)

    try:
        import h5py as hdf
    except ImportError:
        raise ImportError('h5py is required to use HDF5 files')
    
    try:
        assert isinstance(SDobject, SpaceData)
    except AssertionError:
        raise ValueError("Input data is not of type SpaceData, check usage: toHDF5(fname, datamodel)")
    #mash these into a defaults dict...
    if 'mode' not in kwargs:
        wr_mo = 'a'
    else:
        wr_mo = kwargs['mode']
    if 'compression' not in kwargs:
        h5_compr_type = None
    else:
        h5_compr_type = kwargs['compression']
        if h5_compr_type not in ['gzip', 'szip', 'lzf', 'shuffle', 'fletcher32', None]:
            raise NotImplementedError('Specified compression type not supported')
    if ('compression_opts' not in kwargs) or (h5_compr_type == 'lzf'):
        h5_compr_opts = None
    else:
        h5_compr_opts = kwargs['compression_opts']

    if 'overwrite' not in kwargs: kwargs['overwrite'] = True
    if type(fname) in str_classes:
        if os.path.isfile(fname) and not kwargs['overwrite']:
            raise(IOError('Cannot write HDF5, file exists (see overwrite) "{0!s}"'.format(fname)))
        if os.path.isfile(fname) and kwargs['overwrite']:
            os.remove(fname)
        hfile = hdf.File(fname, mode=wr_mo)
        must_close = True
    else:
        hfile = fname
        #should test here for HDF file object
        must_close = False
    if 'path' in kwargs:
        path = kwargs['path']
    else:
        path = '/'

    # long is a type in python2 not in python3
    # unicode is a type in python2 not in python3
    try:
        allowed_attrs = [int, long, float, str, unicode, numpy.ndarray, list, tuple, numpy.string_]
    except NameError:
        allowed_attrs = [int,       float, bytes, str, numpy.ndarray, list, tuple, numpy.string_]
    for v in numpy.typecodes['AllInteger']:
        allowed_attrs.append(numpy.typeDict[v])
    for v in numpy.typecodes['AllFloat']:
        allowed_attrs.append(numpy.typeDict[v])

    allowed_elems = [SpaceData, dmarray]

    #first convert non-string keys to str
    SDobject = convertKeysToStr(SDobject)
    SDcarryattrs(SDobject,hfile,path,allowed_attrs)

    try:
        for key, value in SDobject.items():
            if isinstance(value, allowed_elems[0]):
                hfile[path].create_group(key)
                toHDF5(hfile, SDobject[key], path=path+'/'+key, compression=h5_compr_type, compression_opts=h5_compr_opts)
            elif isinstance(value, allowed_elems[1]):
                try:
                    hfile[path].create_dataset(key, data=value, compression=h5_compr_type, compression_opts=h5_compr_opts)
                except:
                    dumval = value.copy()
                    if isinstance(value[0], datetime.datetime):
                        for i, val in enumerate(value): dumval[i] = val.isoformat()
                    hfile[path].create_dataset(key, data=dumval.astype('|S35'), compression=h5_compr_type, compression_opts=h5_compr_opts)
                    #else:
                    #    hfile[path].create_dataset(key, data=value.astype(float))
                SDcarryattrs(SDobject[key], hfile, path+'/'+key, allowed_attrs)
            else:
                warnings.warn('The following data is not being written as is not of an allowed type\n' +
                               'key = {0} ({1})\n'.format(key, type(key)) +
                                  'value type {0} is not in the allowed data type list'.format(type(value)),
                                      DMWarning)
    finally:
        if must_close:
            hfile.close()


def fromNC3(fname):
    try:
        from scipy.io import netcdf as nc
    except ImportError:
        raise ImportError('SciPy is required to import netcdf3')

    ncfile = nc.netcdf_file(fname, mode='r', mmap=False)

    SDobject = SpaceData(attrs=dmcopy(ncfile._attributes))

    ##carry over the groups and datasets
    for key, value in ncfile.variables.items():
        #try:
            SDobject[key] = dmarray(dmcopy(value.data), attrs=dmcopy(value._attributes))
        #except (TypeError, ZeroDivisionError): #ZeroDivisionError catches zero-sized DataSets
        #    SDobject[key] = dmarray(None)
    ncfile.close()
    return SDobject



def toHTML(fname, SDobject, attrs=(),
           varLinks=False, linkFormat=None, echo=False, tableTag='<table border="1">'):
    """
    Create an HTML dump of the structure of a spacedata

    Parameters
    ----------
    fname : str
        Filename to write to

    SDobject : spacepy.datamodel.SpaceData
        SpaceData with associated attributes and variables in dmarrays

    Other Parameters
    ----------------
    overwrite : bool (optional)
        allow overwrite of an existing target file (default True)
    mode : str (optional)
        HDF5 file open mode (a, w, r) (default 'a')
    echo : bool
        echo the html to the screen
    varLinks : bool
        make the variable name a link to a stub page

    """
    output = StringIO.StringIO() # put the output into a StringIO
    keys = sorted(SDobject.keys())

    output.write(tableTag)
    output.write('\n')
    output.write('<tr><th>{0}</th>'.format('Variable'))
    for attr in attrs:
        output.write('<th>{0}</th>'.format(attr))
    output.write('</tr>')

    for ii, key in enumerate(keys):
        if ii % 2 == 0:
            output.write('<tr>')
        else:
            output.write('<tr class="alt">')
        output.write('<td>')
        if varLinks:
            output.write('<a href="{0}.html">'.format(key))
        output.write('{0}'.format(key))
        if varLinks:
            output.write('</a>')
        output.write('</td>')

        for attr in attrs:
            try:
                if not isinstance(SDobject[key].attrs[attr], (str, unicode)):
                    tmp = str(SDobject[key].attrs[attr])
                    output.write('<td>{0}</td>'.format(_idl2html(tmp)))
                else:
                    output.write('<td>{0}</td>'.format(_idl2html(SDobject[key].attrs[attr])))
            except KeyError:
                output.write('<td></td>')
        output.write('</tr>\n')
    output.write('</table>\n')
    with open(fname, 'w') as fp:
        fp.write(output.getvalue())
    if echo:
        print(output.getvalue())
    output.close()

def _idl2html(idl):
    """
    given an idl format string for text change it to html

    Parameters
    ==========
    idl : str
        idl formated string

    Returns
    =======
    out : str
        html formatted string
    """
    html = idl
    conv = {'!!': '!',
            '!E': '<sup>',
            '!I': '<sub>'}
    while True: # hate it but for now
        ind = html.find('!')
        if ind == -1:
            break
        code = html[ind:ind+2]
        html = html.replace(code, conv[code])
        if code == '!I':
            if '!N' in html:
                html = html.replace('!N', '</sub>', 1 ) # just replace 1
            else:
                html = html + '</sub>'
        elif code == '!E':
            if '!N' in html:
                html = html.replace('!N', '</sup>', 1 ) # just replace 1
            else:
                html = html + '</sup>'
    return html

def readJSONMetadata(fname, **kwargs):
    '''Read JSON metadata from an ASCII data file

    Parameters
    ----------
    fname : str
        Filename to read metadata from

    Other Parameters
    ----------------
    verbose : bool (optional)
        set verbose output so metadata tree prints on read (default False)

    Returns
    -------
    mdata: spacepy.datamodel.SpaceData
        SpaceData with the metadata from the file
    '''
    if hasattr(fname, 'read'):
        lines = fname.read()
    else:
        with open(fname, 'r') as f:
            lines = f.read()

    # isolate header
    p_srch = re.compile(r"^#(.*)$", re.M)
    hreg = re.findall(p_srch, lines)
    header = "".join(hreg)

    # isolate JSON field
    srch = re.search(r'\{\s*(.*)\s*\}', header )
    if isinstance(srch, type(None)):
        raise IOError('The input file has no valid JSON header. Must be valid JSON bounded by braces "{ }".')
    js = srch.group(1)
    inx = js.rfind('end JSON')

    if inx == -1:
        js = ' '.join(('{', js, '}'))
        mdatadict = json.loads(js)
    else:
        js = ' '.join(('{', js[:inx]))
        mdatadict = json.loads(js)

    mdata = SpaceData()
    for key in mdatadict:
       if not hasattr(mdatadict[key], 'keys'): # not dict-like, must be global attrs
           mdata.attrs[key] = mdatadict[key]
       elif 'START_COLUMN' in mdatadict[key]: # is a variable
           mdata[key] = SpaceData(attrs=mdatadict[key])
       elif 'VALUES' in mdatadict[key]: # is global metadata
           dum = mdatadict[key].pop('VALUES')
           mdata[key] = dmarray(dum, attrs=mdatadict[key])
       else: # don't know how to deal with this, store as global attrs
           mdata.attrs[key] = mdatadict[key]

    if 'verbose' in kwargs:
        if kwargs['verbose']:
            mdata.tree(verbose=True, attrs=True)
    return mdata

def readJSONheadedASCII(fname, mdata=None, comment='#', convert=False, restrict=None):
    """read JSON-headed ASCII data files into a SpacePy datamodel

    Parameters
    ----------
    fname : str or list
        Filename(s) to read data from

    Other Parameters
    ----------------
    mdata : spacepy.datamodel.SpaceData (optional)
        supply metadata object, otherwise is read from fname (default None)
    comment: str (optional)
        comment string in file to be read; lines starting with comment are
        ignored (default '#')
    convert: bool or dict-like (optional)
        If True, uses common names to try conversion from string. If a dict-
        like then uses the functions specified as the dict values to convert
        each element of 'key' to a non-string
    restrict: list of strings (optional)
        If present, restrict the variables stored to only those on this list

    Returns
    -------
    mdata: spacepy.datamodel.SpaceData
        SpaceData with the data and metadata from the file
    """
    import dateutil.parser as dup
    filelike = False
    if isinstance(fname, str_classes):
        fname=[fname]
    elif hasattr(fname, 'readlines'):
        fname = [fname]
        filelike = True
    if not mdata:
        mdata = readJSONMetadata(fname[0])
    if restrict:
        delkeys = [kk for kk in mdata.keys() if kk not in restrict]
        for val in delkeys:
            del mdata[val] #remove undesired keys
    mdata_copy = dmcopy(mdata)
    def innerloop(fh, mdata, mdata_copy):
        line = fh.readline()
        if not str is bytes:
            line = line.decode('latin1')
        while (line and line[0]==comment):
            line = fh.readline()
            if not str is bytes:
                line = line.decode('latin1')
        fh.seek(-len(line), os.SEEK_CUR) # fixes the missing first data bug
        alldata = fh.readlines()
        if not alldata:
            return mdata
        if not str is bytes:
            alldata = [d.decode('latin1') for d in alldata]
        ncols = len(alldata[0].rstrip().split())
        # fixes None in the data from empty lines at the end
        for row in range(len(alldata)): # reverse order
            if not alldata[-1].rstrip(): # blank line (or al white space)
                alldata.pop(-1)
            else:
                break
        nrows = len(alldata)
        data = numpy.empty((nrows, ncols), dtype=object)
        for ridx, line in enumerate(alldata):
            for cidx, el in enumerate(line.rstrip().split()):
                data[ridx, cidx] = el
        for key in mdata_copy.keys():
            if 'START_COLUMN' in mdata_copy[key].attrs:
                st = mdata_copy[key].attrs['START_COLUMN']
                if 'DIMENSION' in mdata_copy[key].attrs:
                    varDims = numpy.array(mdata_copy[key].attrs['DIMENSION'])
                    if not varDims.shape: varDims = numpy.array([varDims])
                    singleDim = True
                    if len(varDims)>1 or varDims[0]>1:
                        singleDim = False
                if ('DIMENSION' in mdata_copy[key].attrs) and not singleDim:
                    en = int(mdata_copy[key].attrs['DIMENSION'][0]) + int(st)
                    try:
                        assert mdata[key]=={}
                        mdata[key] = data[:,int(st):int(en)]
                    except (AssertionError, ValueError):
                        mdata[key] = numpy.vstack((mdata[key], data[:,int(st):int(en)]))
                else:
                    try:
                        assert mdata[key]=={}
                        mdata[key] = data[:,int(st)]
                    except (AssertionError, ValueError):
                        mdata[key] = numpy.hstack((mdata[key], data[:,int(st)]))
        return mdata
    for fn in fname:
        if not filelike:
            with open(fn, 'rb') as fh: # fixes windows bug with seek()
                mdata = innerloop(fh, mdata, mdata_copy)
        else:
            mdata = innerloop(fh, mdata, mdata_copy)
    #now add the attributres to the variables
    keys = list(mdata_copy.keys())
    for key in keys:
        if isinstance(mdata[key], SpaceData):
            mdata[key] = dmarray(None, attrs=mdata_copy[key].attrs)
        else:
            mdata[key] = dmarray(mdata[key], attrs=mdata_copy[key].attrs)

    if convert:
        if isinstance(convert, dict):
            conversions=convert
        else:
            conversions = {'DateTime': lambda x: dup.parse(x, ignoretz=True),
                           'ExtModel': lambda x: str(x)}
        for conkey in conversions:
            try:
                name = keys.pop(keys.index(conkey)) #remove from keylist
            except ValueError:
                warnings.warn('Key {0} for conversion not found in file'.format(conkey), UserWarning)
                continue
            for i,element in numpy.ndenumerate(mdata[name]):
                mdata[name][i] = conversions[name](element)

    for remkey in keys:
        try:
            mdata[remkey] = numpy.asanyarray(mdata[remkey], dtype=float)
        except ValueError:
            pass #this will skip any unspecified string fields
    return mdata

def writeJSONMetadata(fname, insd, depend0=None, order=None, verbose=False, returnString=False):
    '''Scrape metadata from SpaceData object and make a JSON header

    Parameters
    ----------
    fname : str
        Filename to write to (can also use a file-like object)
        None can be given in conjunction with the returnString keyword to skip writing output

    insd : spacepy.datamodel.SpaceData
        SpaceData with associated attributes and variables in dmarrays

    Other Parameters
    ----------------
    depend0 : str (optional)
        variable name to use to indicate parameter on which other data depend (e.g. Time)
    order : list (optional)
        list of key names in order of start column in output JSON file
    verbose: bool (optional)
        verbose output
    returnString: bool (optional)
        return JSON header as string instead of returning None

    Returns
    -------
    None  (unless returnString keyword is True)
    '''
    js_out = {}

    def stripNL(text):
        out = text.group().replace('\n','').replace('  ','')
        return out

    #if required, identify depend0 for deciding what's data/metadata
    if depend0 is None:
        #search for DEPEND_0 in metadata
        for key in insd:
            if not hasattr(insd[key], 'attrs'):
                insd[key] = dmarray(insd[key])
            if 'DEPEND_0' in insd[key].attrs:
                depend0 = insd[key].attrs['DEPEND_0']
                if not isinstance(depend0, str_classes):
                    #assume it's a singleton list
                    depend0 = depend0[0]
                if not isinstance(depend0, str_classes):
                    depend0 = None #Failed to get a depend0
                else:
                    break #we're done here
        if depend0 is None:
            #fall back to most common var length
            tmp, keylist = [], list(insd.keys())
            for key in keylist:
                tmp.append(len(insd[key]))
            depend0 = keylist[tmp.index(numpy.bincount(tmp).argmax())]
            #TODO Set using Time, or Epoch, or similar...
    else:
        if not depend0 in insd: raise KeyError('Invalid key supplied for ordering metadata on write')
    datalen = len(insd[depend0])

    #start with global attrs
    #TODO: check for datetime objs in attributes
    if insd.attrs:
        glattr = _dateToISO(insd.attrs)
        for key in glattr:
            js_out[key] = dmcopy(glattr[key])
            #TODO Mark these as global somehow (by omission of some metadata?)
            try:
                js_out[key] = js_out[key].tolist()
            except:
                pass
    #collect keys and put in order for output
    #TODO first check for extant START_COLUMN
    #then check dimensionality so that start column and dims can be added, if not present
    if hasattr(order, '__iter__'):
        keylist = order
        #now make sure that all missing keys are added to end
        for key in sorted(insd.keys()):
            if key not in order: keylist.append(key)
    else:
        ##TODO do we want to have DEPEND0 first in order by default?
        keylist = sorted(insd.keys())

    idx = 0
    for key in keylist:
        js_out[key] = dmcopy(_dateToISO(insd[key].attrs))
        if len(insd[key]) == datalen: #is data
            if verbose: print('data: {0}'.format(key))
            try:
                js_out[key]['DIMENSION'] = list(insd[key].shape[1:])
                if not js_out[key]['DIMENSION']: js_out[key]['DIMENSION'] = [1]
                js_out[key]['START_COLUMN'] = idx
                dims = js_out[key]['DIMENSION']
                idx += int(dims[0])
                if len(dims)>1:
                    l1 = 'The data cannot be properly represented in JSON-headed ASCII as it has too high a rank\n'
                    l2 = 'key = {0} ({1})\n'.format(key, insd[key].shape)
                    l3 = 'Maximum allowed number of dimensions is 2\n'
                    warnings.warn(''.join([l1, l2, l3]), DMWarning)
            except AttributeError: #AttrErr if just metadata
                #js_out[key]['DIMENSION'] = insd[key].attrs['DIMENSION']
                pass
        else: #is metadata
            if verbose: print('metadata: {0}'.format(key))
            js_out[key]['VALUES'] = dmcopy(_dateToISO(insd[key]))
            js_out[key]['DIMENSION'] = [len(js_out[key]['VALUES'])]
        for kk in js_out[key]:
            try:
                js_out[key][kk] = js_out[key][kk].tolist()
            except:
                pass
    json_str = json.dumps(js_out, indent=4, sort_keys=True)
    reob = re.compile(r'\[.*?\]', re.DOTALL)
    json_str = re.sub(reob, stripNL, json_str) #put lists back onto one line
    #add comment field for header
    json_str = ''.join(['#', json_str])
    json_str = '\n#'.join(json_str.split('\n'))
    json_str = ''.join([json_str,'\n'])

    if isinstance(fname, str_classes):
        with open(fname,'w') as fh:
            fh.writelines(json_str)
    elif hasattr(fname, 'writelines'):
        fname.writelines(json_str)
    elif (fname is None) and (returnString):
        return json_str

    if returnString: return json_str


def _dateToISO(indict):
    """
    covert datetimes to iso strings inside of datamodel attributes
    """
    retdict = dmcopy(indict)
    if isinstance(indict, dict):
        for key in indict:
            if isinstance(indict[key], datetime.datetime):
                retdict[key] = retdict[key].isoformat()
            elif hasattr(indict[key], '__iter__'):
                for idx, el in enumerate(indict[key]):
                    if isinstance(el, datetime.datetime):
                        retdict[key][idx] = el.isoformat()
    else:
        if isinstance(indict, datetime.datetime):
            retdict = retdict.isoformat()
        elif hasattr(indict, '__iter__'):
            retdict = numpy.asanyarray(retdict)
            for idx, el in numpy.ndenumerate(indict):
                if isinstance(el, datetime.datetime):
                    retdict[idx] = el.isoformat()
    return retdict


def toJSONheadedASCII(fname, insd, metadata=None, depend0=None, order=None, **kwargs):
    '''Write JSON-headed ASCII file of data with metadata from SpaceData object

    Parameters
    ----------
    fname : str
        Filename to write to (can also use a file-like object)
        None can be given in conjunction with the returnString keyword to skip writing output

    insd : spacepy.datamodel.SpaceData
        SpaceData with associated attributes and variables in dmarrays

    Other Parameters
    ----------------
    depend0 : str (optional)
        variable name to use to indicate parameter on which other data depend (e.g. Time)
    order : list (optional)
        list of key names in order of start column in output JSON file
    metadata: str or file-like (optional)
        filename with JSON header to use (or file-like with JSON metadata)
    delimiter: str
        delimiter to use in ASCII output (default is whitespace), for tab, use '\t'

    Returns
    -------
    None

    Examples
    --------
    >>> import spacepy.datamodel as dm
    >>> data = dm.SpaceData()
    >>> data.attrs['Global'] = 'A global attribute'
    >>> data['Var1'] = dm.dmarray([1,2,3,4,5], attrs={'Local1': 'A local attribute'})
    >>> data['Var2'] = dm.dmarray([[8,9],[9,1],[3,4],[8,9],[7,8]])
    >>> data['MVar'] = dm.dmarray([7.8], attrs={'Note': 'Metadata'})
    >>> dm.toJSONheadedASCII('outFile.txt', data, depend0='Var1', order=['Var1'])
    #Note that not all field names are required, those not given will be listed
    #alphabetically after those that are specified
    '''
    kwarg_dict = {'delimiter': ' '}
    for key in kwarg_dict:
        if key in kwargs:
            kwarg_dict[key] = kwargs[key]
    if not metadata:
        metadata = StringIO.StringIO()
        writeJSONMetadata(metadata, insd, depend0=depend0, order=order)
        metadata.seek(0) #rewind StringIO object to start
    hdr = readJSONMetadata(metadata)

    datlist = []
    for key in hdr:
        if 'START_COLUMN' in hdr[key].attrs:
            #add to list of (start_col, keyname) pairs
            datlist.append((hdr[key].attrs['START_COLUMN'], key, hdr[key].attrs['DIMENSION'][0]))
            #also use for data length
            datlen = len(insd[key])
            if datlen==0:
                raise ValueError('No data present to write: Use writeJSONmetadata')
                #TODO: Set this to just default to writing the header out and raise a warning
    datlist.sort()
    ncols = datlist[-1][0]+datlist[-1][-1]

    #now open file (file-like) and for each line in len(data)
    #write the line using start_column, name, dimension
    data = numpy.zeros([datlen,ncols], dtype=object)
    for stcol, name, dim in datlist:
        if dim==1:
            data[:, stcol] = _dateToISO(insd[name])
        else:
            data[:, stcol:stcol+dim] = _dateToISO(insd[name])
    hdstr = writeJSONMetadata(None, hdr, depend0=depend0, order=order, returnString=True)
    with open(fname, 'w') as fh:
        fh.writelines(hdstr)
        for line in data:
            prline = kwarg_dict['delimiter'].join([str(el) for el in line])
            fh.write(''.join([prline,'\n']))


def fromRecArray(recarr):
    '''Takes a numpy recarray and returns each field as a dmarray in a SpaceData container
    
    Parameters
    ----------
    recarr : numpy record array
        object to parse into SpaceData container

    Returns
    -------
    sd: spacepy.datamodel.SpaceData
        dict-like containing arrays of named records in recarr

    Examples
    --------
    >>> import numpy as np
    >>> import spacepy.datamodel as dm
    >>> x = np.array([(1.0, 2), (3.0, 4)], dtype=[('x', float), ('y', int)])
    >>> print(x, x.dtype)
    array([(1.0, 2), (3.0, 4)], dtype=[('x', '<f8'), ('y', '<i4')])
    >>> sd = dm.fromRecArray(x)
    >>> sd.tree(verbose=1)
    +
    |____x (spacepy.datamodel.dmarray (2,))
    |____y (spacepy.datamodel.dmarray (2,))
    '''
    sd = SpaceData()
    for key in recarr.dtype.fields.keys():
        sd[key] = dmarray(recarr[key])
    return sd

def toRecArray(sdo):
    '''Takes a SpaceData and creates a numpy recarray

    Parameters
    ----------
    sdo : SpaceData
        SpaceData to change to a numpy recarray

    Returns
    -------
    recarr: numpy record array
        numpy.recarray object with the same values (attributes are lost)

    Examples
    --------
    >>> import numpy as np
    >>> import spacepy.datamodel as dm
    >>> sd = dm.SpaceData()
    >>> sd['x'] = dm.dmarray([1.0, 2.0])
    >>> sd['y'] = dm.dmarray([2,4])
    >>> sd.tree(verbose=1)
    +
    |____x (spacepy.datamodel.dmarray (2,))
    |____y (spacepy.datamodel.dmarray (2,))
    >>> ra = dm.toRecArray(sd)
    >>> print(ra, ra.dtype)
    [(2, 1.0) (4, 2.0)] (numpy.record, [('y', '<i8'), ('x', '<f8')])
    '''
    nametype = numpy.dtype([(k, sdo[k].dtype.str) for k in sdo])
    recarr = numpy.rec.fromarrays( [sdo[k] for k in sdo], dtype=nametype)
    return recarr


def dmcopy(dobj):
    '''Generic copy utility to return a copy of a (datamodel) object

    Parameters
    ----------
    dobj : object
        object to return a copy of

    Returns
    -------
    copy_obj: object (same type as input)
        copy of input oibject

    Examples
    --------
    >>> import spacepy.datamodel as dm
    >>> dat = dm.dmarray([2,3], attrs={'units': 'T'})
    >>> dat1 = dm.dmcopy(dat)
    >>> dat1.attrs['copy': True]
    >>> dat is dat1
    False
    >>> dat1.attrs
    {'copy': True, 'units': 'T'}
    >>> dat.attrs
    {'units': 'T'}
    '''
    if isinstance(dobj, (SpaceData, dmarray)):
        return copy.deepcopy(dobj)
    elif isinstance(dobj, numpy.ndarray):
        return numpy.copy(dobj)
    else:
        return copy.copy(dobj)

def createISTPattrs(datatype, ndims=1, vartype=None, units=' ', NRV=False):
    '''Return set of unpopulated attributes for ISTP compliant variable

    Parameters
    ----------
    datatype : {'data', 'support_data', 'metadata'}
        datatype of variable to create metadata for.
    ndims : int
        number of dimensions, default=1
    vartype : {'float', 'char', 'int', 'epoch', 'tt2000'}
        The type of the variable, default=float
    units : str
        The units of the variable, default=' '
    NRV : bool
        Is the variable NRV (non-record varying), default=False

    Returns
    -------
    attrs : dict
        dictionary of attributes for the variable

    Examples
    --------
    >>> import spacepy.datamodel as dm
    >>> dm.createISTPattrs('data', ndims=2, vartype='float', units='MeV')
    {'CATDESC': '',
     'DISPLAY_TYPE': 'spectrogram',
     'FIELDNAM': '',
     'FILLVAL': -1e+31,
     'FORMAT': 'F18.6',
     'LABLAXIS': '',
     'SI_CONVERSION': ' > ',
     'UNITS': 'MeV',
     'VALIDMIN': '',
     'VALIDMAX': '',
     'VAR_TYPE': 'data',
     'DEPEND_0': 'Epoch',
     'DEPEND_1': ''}
    '''
    fillvals = {'float': -1e31,
                'char': '',
                'int': numpy.array(-2147483648).astype(numpy.int32),
                'epoch': -1.0E31, #datetime.datetime(9999,12,31,23,59,59,999)}
                'tt2000': numpy.array(-9223372036854775808).astype(numpy.int64)}
    formats = {'float': 'F18.6',
               'char': 'A30',
               'int': 'I11',
               'epoch': '',
               'tt2000': 'I21'}
    disp = {1: 'time_series',
            2: 'spectrogram',
            3: 'spectrogram',
            4: 'spectrogram'}
    if vartype not in fillvals:
        fill = -1e31
        form = 'F15.6'
    else:
        fill = fillvals[vartype]
        form = formats[vartype]

    unit = units

    if datatype == 'data':
        attrs = {'CATDESC': '',
            'DISPLAY_TYPE': disp[ndims],
            'FIELDNAM': '',
            'FILLVAL': fill,
            'FORMAT': form,
            'LABLAXIS': '',
            'SI_CONVERSION': ' > ',
            'UNITS': unit,
            'VALIDMIN': '',
            'VALIDMAX': '',
            'VAR_TYPE': 'data'
            }
        for dim in range(ndims):
            attrs['DEPEND_{0}'.format(dim)] = ''
        attrs['DEPEND_0'] = 'Epoch'
    elif datatype == 'support_data':
        attrs = {'CATDESC': '',
            'FIELDNAM': '',
            'FORMAT': form,
            'UNITS': unit,
            'VAR_TYPE': 'support_data'
            }
        for dim in range(ndims):
            attrs['DEPEND_{0}'.format(dim)] = ''
        if not NRV:
            attrs['VALIDMIN'] = ''
            attrs['VALIDMAX'] = ''
            attrs['FILLVAL'] = fill
            attrs['DEPEND_0'] = 'Epoch'
        else:
            del attrs['DEPEND_0']
    elif datatype == 'metadata':
        attrs = {'CATDESC': '',
            'FIELDNAM': '',
            'FORMAT': form,
            'UNITS': unit,
            'VAR_TYPE': 'metadata'
            }
        for dim in range(ndims):
            attrs['DEPEND_{0}'.format(dim)] = ''
        if not NRV:
            attrs['FILLVAL'] = fill
            attrs['DEPEND_0'] = 'Epoch'
        else:
            del attrs['DEPEND_0']
    else:
        raise ValueError("Invalid datatype (data|support_data|metadata)")

    return attrs

def _getVarLengths(data):
    """
    get the length of all the variables
    
    Parameters
    ----------
    data : SpaceData
        SpaceData object to return the length of the variables

    Returns
    -------
    data : dict
        dict of the names and lengths of a SpaceData
    """
    ans = {}
    for k, v in data.items():
        ans[k] = len(v)
    return ans

def resample(data, time=[], winsize=0, overlap=0, st_time=None, outtimename='Epoch'):
    """
    resample a SpaceData to a new time interval

    Parameters
    ----------
    data : SpaceData or dmarray
        SpaceData with data to resample or dmarray with data to resample,
        variables can only be 1d or 2d, if time is specified only variables
        the same length as time are resampled, otherwise only variables
        with length equal to the longest length are resampled

    time : array-like
        dmarray of times the correspond to the data

    winsize : datetime.timedelta
        Time frame to average the data over

    overlap : datetime.timedelta
        Overlap in the moving average

    st_time : datetime.datetime
        Starting time for the resample, if not specified the time of the first
        data point is used (see spacepy.toolbox.windowMean)

    Returns
    -------
    ans : SpaceData 
        Resampled data, included keys are in the input keys (with the data caveats above)
        and Epoch which contains the output time

    Examples
    --------
    >>> import datetime
    >>> import spacepy.datamodel as dm
    >>> a = dm.SpaceData()
    >>> a.attrs['foo'] = 'bar'
    >>> a['a'] = dm.dmarray(range(10*2)).reshape(10,2)
    >>> a['b'] = dm.dmarray(range(10)) + 4
    >>> a['c'] = dm.dmarray(range(3)) + 10
    >>> times = [datetime.datetime(2010, 1, 1) + datetime.timedelta(hours=i) for i in range(10)]
    >>> out = dm.resample(a, times, winsize=datetime.timedelta(hours=2), overlap=datetime.timedelta(hours=0))
    >>> out.tree(verbose=1, attrs=1)
    # +
    # :|____foo (str [3])
    # |____Epoch (spacepy.datamodel.dmarray (4,))
    # |____a (spacepy.datamodel.dmarray (4, 2))
    # :|____DEPEND_0 (str [5])
    #
    # Things to note:
    #    - attributes are preserved
    #    - the output variables have their DEPEND_0 changed to Epoch (or outtimename)
    #    - each dimension of a 2d array is resampled individually 
    """
    from . import toolbox
    # check for SpaceData or dmarray input before going to a bunch of work
    if not isinstance(data, (SpaceData, dmarray)):
        raise(TypeError('Input must be a SpaceData or dmarray object'))

    # can only resample variables that have the same length as time,
    #    if time is default then use all the vals that are the same
    #    as the longest var
    lent = len(time)
    if lent == 0: lent = len(data[max(data, key=lambda k: len(data[k]))])
    keys = [k for k in data if len(data[k]) == lent]
    d2 = data[keys]
    # what time are we starting at?
    try:
        t_int = time.UTC
    except AttributeError:
        t_int = dmarray(time)
    if t_int.any() and ((st_time is None) and isinstance(t_int[0], datetime.datetime)):
        st_time = t_int[0].replace(hour=0, minute=0, second=0, microsecond=0)

    ans = SpaceData()
    ans.attrs = data.attrs
    
    for k in keys:
        if len(data[k].shape) > 1:
            if len(data[k].shape) > 2:
                raise(IndexError("Variables can only be 1d or 2d"))
            for i in range(data[k].shape[1]):
                d, t = toolbox.windowMean(data[k][:,i], time=t_int, winsize=winsize, overlap=overlap, st_time=st_time)
                if k not in ans:
                    ans[k] = dmarray(d)
                else:
                    ans[k] = dmarray.vstack(ans[k], d)
            ans[k] = ans[k].T
        else:
            d, t = toolbox.windowMean(data[k], time=t_int, winsize=winsize, overlap=overlap, st_time=st_time)
            ans[k] = dmarray(d)
        try:
            ans[k].attrs = data[k].attrs
        except AttributeError: # was not a dmarray
            pass
        ans[k].attrs['DEPEND_0'] = outtimename
    ans[outtimename] = dmarray(t)
        
    return ans


    
