Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Normal
Fix Version/s: RHODS_1.27.0_GA
Affects Version/s: None
Component/s: Pipelines
Labels:
- MLOps
- groomed

Story Points:
2
Blocked:
False
Blocked Reason:
None
Ready:
False
Affects Testing:

Testable
Automated:
No
CDW blocker:
CDW devel_ack:
CDW docs_ack:
CDW pm_ack:
CDW qa_ack:
CDW release:
Fixed in Build:
1.27.0-24
Regression:
No
Target Release:

RHODS_1.27.0_GA
Test Blocker:
No
Test Coverage:

N/A
Watchlist Impact:
None
Git Pull Request:
https://github.com/opendatahub-io/data-science-pipelines-operator/pull/106
Intelligence Requested:
Market:

Sprint:
ML Ops Sprint 1.26, ML Ops Sprint 1.27

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

During sandbox performance tests, we noticed that there is an idler task that scale down all Deployments to save resources. For the MariaDB deployment, it keeps trying to create a new pod, putting in an endless loop with the idler terminating the pod until we get the InnoDB files corrupted. In another scenario, the frequent MariaDB pod crashes end in a corruption in the data files.

We need to find out how to avoid that scenario, not only for sandbox clusters but for production cluster.

Error we encounter:

=> sourcing 20-validate-variables.sh ...
=> sourcing 25-validate-replication-variables.sh ...
=> sourcing 30-base-config.sh ...
---> 19:35:18 Processing basic MySQL configuration files ...
=> sourcing 60-replication-config.sh ...
=> sourcing 70-s2i-config.sh ...
---> 19:35:18 Processing additional arbitrary MySQL configuration provided by s2i ...
=> sourcing 40-paas.cnf ...
=> sourcing 50-my-tuning.cnf ...
---> 19:35:18 Starting MySQL server with disabled networking ...
---> 19:35:18 Waiting for MySQL to start ...
2023-03-28 19:35:18 0 [Note] /usr/libexec/mysqld (mysqld 10.3.35-MariaDB) starting as process 25 ...
2023-03-28 19:35:18 0 [Note] InnoDB: Using Linux native AIO
2023-03-28 19:35:18 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-03-28 19:35:18 0 [Note] InnoDB: Uses event mutexes
2023-03-28 19:35:18 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-03-28 19:35:18 0 [Note] InnoDB: Number of pools: 1
2023-03-28 19:35:18 0 [Note] InnoDB: Using SSE2 crc32 instructions
2023-03-28 19:35:18 0 [Note] InnoDB: Initializing buffer pool, total size = 512M, instances = 1, chunk size = 128M
2023-03-28 19:35:18 0 [Note] InnoDB: Completed initialization of buffer pool
2023-03-28 19:35:18 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-03-28 19:35:18 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1625470
2023-03-28 19:35:18 0 [ERROR] InnoDB: Missing MLOG_CHECKPOINT at 1625470 between the checkpoint 1625470 and the end 1625488.
2023-03-28 19:35:18 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-03-28 19:35:19 0 [Note] InnoDB: Starting shutdown...
2023-03-28 19:35:19 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-03-28 19:35:19 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-03-28 19:35:19 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-28 19:35:19 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2023-03-28 19:35:19 0 [ERROR] Aborting