jimfuqian/BB2-3641 Added 2 Django Commands to repair and set expiration date of grants

[JFU-NAVA-PBC]

JIRA Ticket:
BB2-3641

What Does This PR Do?

See 'Context" in jira ticket for details.

Some grants are impacted - their expiration_date now have incorrect values:

case1: the impacted grants have NULL in their expiration_date
case2: the impacted grants have (mostly) their expiration_date previous date extended

Added Django command ( repair_grants_expiration_date ) to repair the grants:

What Should Reviewers Watch For?

The commands can be tested locally, the repair details are described in ticket
(commands for internal use)

If you're reviewing this PR, please check for these things in particular:

Validation

Steps to test locally:

Django command: repair_grants_expiration_date:

1. Check out PR
2. Spin up a local bb2 server(docker compose)
3. docker ps to identify the bb2 server container name, and bash into the bb2 docker
4. Run
5. Generate fair amount of sample data: e.g. Run: python manage.py create_test_users_and_applications_batch -b 1000 -d 5 -a 5
6. Check the grants generated e.g. by SQL the grants table:

8. Take notes of the grants associated with the app you want to test the new commands with
9. Run help of the command to see the syntax
10. FIX case1: do DRYRUN to preview result:
python manage.py repair_grants_expiration_date --dryrun --appname 'app0_DevUserFN4.DevUserLN4' --appnameagain 'app0_DevUserFN4.DevUserLN4' --range '01/21/25 03:34:12-01/21/25 03:34:16' Info: dryrun = True, this is a dry run, no change to database. process restriction: feature turned on date: None and created_at in range: [2025-01-21 03:34:12+00:00, 2025-01-21 03:34:16+00:00]. Number of grants to be checked for repair = 26 +#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+# Processed grants: 26, created after feature turned on date: 26, created before turned on date: 0
11. Check DRYRUN output, do a real run if preview looks good:
python manage.py repair_grants_expiration_date --appname 'app0_DevUserFN4.DevUserLN4' --appnameagain 'app0_DevUserFN4.DevUserLN4' --range '01/21/25 03:34:12-01/21/25 03:34:16' process restriction: feature turned on date: None and created_at in range: [2025-01-21 03:34:12+00:00, 2025-01-21 03:34:16+00:00]. Number of grants to be checked for repair = 26 +$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$ Processed grants: 26, created after feature turned on date: 26, created before turned on date: 0
12. Query the table authorization_dataaccessgrant for the grants (for app: app0_DevUserFN4.DevUserLN4), verify that the grants in the date time range have their expiration_date set to their created_at + 13month.
13. FIX case2 (--turnondate present): do DRYRUN
python manage.py repair_grants_expiration_date --dryrun --appname 'app0_DevUserFN4.DevUserLN4' --appnameagain 'app0_DevUserFN4.DevUserLN4' --range '01/21/25 03:34:11-01/21/25 03:34:17' --turnondate '01/21/25 03:34:15' Info: dryrun = True, this is a dry run, no change to database. process restriction: feature turned on date: 2025-01-21 03:34:15+00:00 and created_at in range: [2025-01-21 03:34:11+00:00, 2025-01-21 03:34:17+00:00]. Number of grants to be checked for repair = 50 -#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#+#-#+# Processed grants: 50, created after feature turned on date: 20, created before turned on date: 30
14. Check DRYRUN output, do a real run if preview looks good:
python manage.py repair_grants_expiration_date --appname 'app0_DevUserFN4.DevUserLN4' --appnameagain 'app0_DevUserFN4.DevUserLN4' --range '01/21/25 03:34:11-01/21/25 03:34:17' --turnondate '01/21/25 03:34:15' process restriction: feature turned on date: 2025-01-21 03:34:15+00:00 and created_at in range: [2025-01-21 03:34:11+00:00, 2025-01-21 03:34:17+00:00]. Number of grants to be checked for repair = 50 -$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$-$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$+$-$+$ Processed grants: 50, created after feature turned on date: 20, created before turned on date: 30
15. Query the table authorization_dataaccessgrant for the grants (for app: app0_DevUserFN4.DevUserLN4), verify that the grants in the date time range (50) have their expiration_date set to their created_at + 13month if the created_at is after the turn_on_date (20), otherwise the expiration_date is set to turn_on_date + 13month (30).

What Security Implications Does This PR Have?

Please indicate if this PR does any of the following:

• Adds any new software dependencies
• Modifies any security controls
• Adds new transmission or storage of data
• Any other changes that could possibly affect security?

• Yes, one or more of the above security implications apply. This PR must not be merged without the ISSO or team
  security engineer's approval.

Any Migrations?

• Yes, there are migrations
  • The migrations should be run PRIOR to the code being deployed
  • The migrations should be run AFTER the code is deployed
  • There is a more complicated migration plan (downtime,
    etc)
• No migrations

[jimmyfagan]

Have some comments, maybe I'm being too nitpicky given that this is code that we will probably only use for the next week or so and then eventually remove, but I think there's a lot of ways we can make this code cleaner, so let me know what you think! I'm available to pair on this tomorrow if that helps too.

[jimmyfagan]

I don't understand the reason for this restriction, the created before/after seems to just be used to determine which grants to update, and the turnondate seems to only be used to determine what the new expiration date should be, so it would still be valid to use those parameters, even if the turnondate is outside of that range.

[jimmyfagan]

Is this case possible given the check above?

[JFU-NAVA-PBC]

likely not, just checking (paranoid)

[jimmyfagan]

This is where the function actually starts doing something interesting, over 100 lines into the function. Can we use helper methods for all of the validation above to make this easier to read? Something like validate_inputs that does all of that in a separate function, and/or something like validate_and_retrieve_date_arg that can be used to reduce some of the duplicated error checking code.

[JFU-NAVA-PBC]

yea, lots a validation and core code is a few lines

[jimmyfagan]

Is there a reason to terminate when encountering a null expiration date? Seems like we would just want to handle that by setting it to 13 months past the creation date.

[JFU-NAVA-PBC]

in the use case dealing with, a null is unexpected, so prefer to stop and check....

[jimmyfagan]

A lot of the comments from the other file apply here as well. One thought that comes to mind, since these functions are so similar, is that we could simplify testing/review/etc to have one function that handles both of these cases. Basically it would set the expiration to 13 months after creation if its expiration is not already set; if the expiration is already set, then it would not change anything if the turnondate isn't set, or it would change it to 13 months after creation date or turnondate based on the current logic in repair_grants_expiration date. I won't insist on it, just think it simplifies a lot of this.

[JFU-GIT]

Thx for the feedback, changes are on the way

[jimmyfagan]

New code looks good! However I am having trouble testing it out:

$ python manage.py repair_grants_expiration_date --dryrun --appname 'app0_DevUserFN0.DevUserLN0' --appnameagain 'app0_DevUserFN0.DevUserLN0' --range '01/16/25 19:58:26-01/18/25 23:59:59'
Error: bad date time value: 01/16/25 19:58:26
time data '01/16/25 19:58:26' does not match format '%m/%d/%Y %H:%M:%S'

Am I doing something wrong, or is there an issue with the date format string?

[JFU-GIT]

Oh use 2025

[jimmyfagan]

Looks good, I like the adjustments to the summary print statements as well.