Allow scope new for arrays for dynamic stack allocation

[rikkimax]

alloca is typically implemented as an intrinsic.

But as a function prototype it is inherently unsafe without an additional attribute to inform the caller that it must not escape up or out of the call stack.

However we have a way to deal with this, by using new and putting it into a scope variable:

void main() @nogc {
    size_t val = 2;
    scope int[] v = new int[val];
}

Currently this does not work for arrays, but does for classes.

Expanding new to work for other types, would be a memory safe method that does not need this unsafe function prototype that we could then remove at a later date.

[ZILtoid1991]

I think alloca() could be kept in some way or form, as a C interop/porting option, while obviously disallowing even in @trusted code if possible, but that would possibly add to the complexity on top of the 3 level memory-safety system.

[Herringway]

void foo(int bar) @nogc {
  scope int[] a = new int[](bar);
  a ~= 42;
}

Error? Allowed?
And if that's allowed, what about:

void foo(int bar) @nogc {
  scope int[] a = new int[](bar);
  scope int[] b = new int[](bar);
  a ~= 42;
}

These could be complicated, but they seem possible.

[rikkimax]

That won't work, because ~= is a GC operation.

To do this with solely stack memory would require value range propagation which is a form of data flow analysis, which unfortunately would be specific to this use case.

[Herringway]

That won't work, because ~= is a GC operation.

So is new T[].

To do this with solely stack memory would require value range propagation which is a form of data flow analysis, which unfortunately would be specific to this use case.

I don't know what that is, but I'm assuming that it means this won't ever get implemented?

[rikkimax]

That won't work, because ~= is a GC operation.

So is new T[].

When scope is on the variable, it effectively is alloca. This is implemented for classes, just not for other types.

So no, not GC. It would be a stack allocation.

void main() @nogc {
    int[] heap = new int(2); // Error
    scope int[] stack = new int(2); // Ok
}

To do this with solely stack memory would require value range propagation which is a form of data flow analysis, which unfortunately would be specific to this use case.

I don't know what that is, but I'm assuming that it means this won't ever get implemented?

Too low of value, and too high of a cost. So yeah that's a big no.

[thewilsonator]

alloca is an intrinsic, you can't deprecate that and definitely can't remove that. LDC (and I presume GDC) have their own version of it too. Adjusted title accordingly.

[rikkimax]

https://github.com/ldc-developers/ldc/blob/c71848f961059d7aa5fc67220017227fa3ec8dc3/runtime/druntime/src/core/stdc/stdlib.d#L229

We can deprecate the function, internally it could be kept.

It is also @system, at the very least we can recommend against using alloca by a comment. This makes me wonder if it would be good to have a way to hint that a symbol shouldn't be used, rather than a deprecation for tooling to suggest.

[dkorpel]

Two key properties of class instances that make stack allocation viable:

• they have a compile-time known size
• they're typically small

What if I allocate something that exceeds the stack size?

void main() @nogc
{
    scope int[] arr = new int[32_000_000];
}

[Geod24]

Regarding appending: It should behave as appending to a slice of a static array. Which currently leads to reallocation. I don't think we'd need to do something for it to work, since the GC will not just be able to find a block for it, as currently happens for static array.

What if I allocate something that exceeds the stack size?

Can always happen, can it ? E.g. you can have @safe code that corrupts memory because, well, fibers. You can also blow up your executable size by having a 128 Mb static array in a class.
scope-allocated dynamic size arrays are pretty high on my list of wanted feature, as they make a lot of allocating patterns go away.

[pbackus]

I thought D was moving away from DIP1000/escape analysis, but if it's not, this is a pretty logical feature to build on top of that capability.

Obviously if -preview=dip1000 or equivalent isn't enabled, this has to be @system.

[dkorpel]

Can always happen, can it ? E.g. you can have @safe code that corrupts memory because, well, fibers.

Sounds like a bug in fibers, either they should probe guard pages, or be @system.

You can also blow up your executable size by having a 128 Mb static array in a class.

I'm not talking about executable size, the problem is that it's normal to allocate multiple megabytes with new and expect it to succeed, unless you're out of RAM. Changing that to stack allocation causes unnecessary runtime crashes.

[Herringway]

Can always happen, can it ? E.g. you can have @safe code that corrupts memory because, well, fibers.

Sounds like a bug in fibers, either they should probe guard pages, or be @system.

You can also blow up your executable size by having a 128 Mb static array in a class.

I'm not talking about executable size, the problem is that it's normal to allocate multiple megabytes with new and expect it to succeed, unless you're out of RAM. Changing that to stack allocation causes unnecessary runtime crashes.

You probably shouldn't change those to stack allocations, then.

void main() {
  int[32_000_000] foo; // segfault ahoy
}

This is already legal and crashes just as spectacularly as you'd expect.

[dkorpel]

You probably shouldn't change those to stack allocations, then.

Exactly. So my question is, how (in this proposal) does the compiler decide whether new int[n] with runtime n fits on the stack or not? And if it doesn't, what does it fall back to: the GC? malloc + free?

[rikkimax]

You probably shouldn't change those to stack allocations, then.

Exactly. So my question is, how (in this proposal) does the compiler decide whether new int[n] with runtime n fits on the stack or not? And if it doesn't, what does it fall back to: the GC? malloc + free?

1. Opt-in with scope (if it wasn't then we'd need to make decisions).
2. With a static array the compiler doesn't protect you from stack overflowing.
   Even if we modelled it at CT, it wouldn't know the stack size.

As it currently stands it won't compile in @nogc to new an array. So its safe to enable that there.

But what to do about the other functions? That may need to wait for an edition.

[Herringway]

You probably shouldn't change those to stack allocations, then.

Exactly. So my question is, how (in this proposal) does the compiler decide whether new int[n] with runtime n fits on the stack or not? And if it doesn't, what does it fall back to: the GC? malloc + free?

1. Opt-in with scope (if it wasn't then we'd need to make decisions).
2. With a static array the compiler doesn't protect you from stack overflowing.
   Even if we modelled it at CT, it wouldn't know the stack size.

As it currently stands it won't compile in @nogc to new an array. So its safe to enable that there.

But what to do about the other functions? That may need to wait for an edition.

What other functions? malloc and alloca? They cannot be removed. Every D compiler out there requires them in order to compile C in whatever form they support doing so. At best, they'd cease to be exposed to D, which makes porting C to D more annoying. Are there some hidden advantages to this?

D doesn't have editions, so waiting for that seems like a bad idea.

[rikkimax]

What other functions? malloc and alloca?

I was referring to functions not marked @nogc.

D doesn't have editions, so waiting for that seems like a bad idea.

We can remove or replace the check for @nogc later on. Which is an improvement if we go in that direction.

No need to solve everything all at once.