casting from datetime64

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

casting from datetime64

Alex Samuel
Hi,

When registering a custom cast function from datetime64 to another dtype, how can I get the units?

I am calling PyArray_RegisterCastFunc from NPY_DATETIME.  Ideally, I'd like to register a separate cast function for each datetime64 units (or none at all... I don't want all units to be castable).  Next best thing would be to obtain the units in the cast function and dispatch accordingly.

Is this possible?  I glanced through the code, and it looks like there's a lot of hard-coded logic around datetime64, but I didn't go through it carefully.  Thought I'd ask before drilling further down.

Thanks in advance,
Alex



_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: casting from datetime64

Sebastian Berg
Hi Alex,

On Mon, 2019-05-13 at 00:35 -0400, Alex Samuel wrote:

> Hi,
>
> When registering a custom cast function from datetime64 to another
> dtype, how can I get the units?
>
> I am calling PyArray_RegisterCastFunc from NPY_DATETIME.  Ideally,
> I'd like to register a separate cast function for each datetime64
> units (or none at all... I don't want all units to be castable).
> Next best thing would be to obtain the units in the cast function and
> dispatch accordingly.
>
> Is this possible?  I glanced through the code, and it looks like
> there's a lot of hard-coded logic around datetime64, but I didn't go
> through it carefully.  Thought I'd ask before drilling further down.
>
No, I do not think that is possible. But you do get the array pointers
during the cast. I honestly would prefer not to promise that all of
this will survive if we change this in numpy, since there are almost no
users. But I think if we change it I could promise to help with
cleaning it up in ora.

I think this is public API (but I do not really like anyone using it
;)), so you can use:

```
/*
 * This function returns a pointer to the DateTimeMetaData
 * contained within the provided datetime dtype.
 */
static PyArray_DatetimeMetaData *
get_datetime_metadata_from_dtype(PyArray_Descr *dtype)
{
    /* original error check for DATETIME unnecessary for you */
    return &(((PyArray_DatetimeDTypeMetaData *)dtype->c_metadata)->meta);
}
```

And in the castfunc (fromarr is passed in as a void * as far as):
```
NPY_DATETIMEUNIT base = get_datetime_metadata_from_dtype(
        PyArray_DESCR(fromarr))->base;
```

Where NPY_DATETIMEUNIT is the enum defined in ndarraytypes.h. The logic
will have to happen inside the cast func.

I would hope there was a better way, but I cannot think of any, and I
am scared that supporting this will add yet another ugly hack when we
want to improve dtypes...

Best,

Sebastian



> Thanks in advance,
> Alex
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: casting from datetime64

Alex Samuel
Thanks!  I understand your hesitation; certainly if I am using undocumented APIs (or even if they are otherwise unused) I understand that it will be on my shoulders to preserve compatibility if you change things.  But I would definitely like to get basic functionality working with current (and older, to be honest) NumPy versions.

Ah, I missed previously that the array objects are passed as the void* args to the PyArray_VectorUnaryFunc.  But it's documented here.
(Why aren't these two args PyArrayObject* then?)

As far as I can tell, get_datetime_metadata_from_dtype() isn't a public API?  It certainly isn't spelled like one.  It's declared in numpy/core/src/multiarray/_datetime.h, which doesn't appear to be #included in any other header.  I suppose I could duplicate the struct layout in my own code to fish out the field I need.  Obviously, compatibility is my own problem then.

Regards,
Alex



On Mon, May 13, 2019, at 01:31, Sebastian Berg wrote:
Hi Alex,

On Mon, 2019-05-13 at 00:35 -0400, Alex Samuel wrote:
> Hi,

> When registering a custom cast function from datetime64 to another
> dtype, how can I get the units?

> I am calling PyArray_RegisterCastFunc from NPY_DATETIME.  Ideally,
> I'd like to register a separate cast function for each datetime64
> units (or none at all... I don't want all units to be castable). 
> Next best thing would be to obtain the units in the cast function and
> dispatch accordingly.

> Is this possible?  I glanced through the code, and it looks like
> there's a lot of hard-coded logic around datetime64, but I didn't go
> through it carefully.  Thought I'd ask before drilling further down.
> get_datetime_metadata_from_dtype

No, I do not think that is possible. But you do get the array pointers
during the cast. I honestly would prefer not to promise that all of
this will survive if we change this in numpy, since there are almost no
users. But I think if we change it I could promise to help with
cleaning it up in ora.

I think this is public API (but I do not really like anyone using it
;)), so you can use:

```
/*
* This function returns a pointer to the DateTimeMetaData
* contained within the provided datetime dtype.
*/
static PyArray_DatetimeMetaData *
get_datetime_metadata_from_dtype(PyArray_Descr *dtype)
{
    /* original error check for DATETIME unnecessary for you */
    return &(((PyArray_DatetimeDTypeMetaData *)dtype->c_metadata)->meta);
}
```

And in the castfunc (fromarr is passed in as a void * as far as):
```
NPY_DATETIMEUNIT base = get_datetime_metadata_from_dtype(
        PyArray_DESCR(fromarr))->base;
```

Where NPY_DATETIMEUNIT is the enum defined in ndarraytypes.h. The logic
will have to happen inside the cast func.

I would hope there was a better way, but I cannot think of any, and I
am scared that supporting this will add yet another ugly hack when we
want to improve dtypes...

Best,

Sebastian



> Thanks in advance,
> Alex


> _______________________________________________
> NumPy-Discussion mailing list
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
https://mail.python.org/mailman/listinfo/numpy-discussion


Attachments:
  • signature.asc


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion