From 590e43c8273f018abb9800243350482dfebcd083 Mon Sep 17 00:00:00 2001 From: Zeedif Date: Tue, 19 Aug 2025 13:00:19 -0600 Subject: [PATCH] feat(sync/koreader): Add KOReader reading progress synchronization (#1560) * feat(sync/koreader): implement reading progress synchronization This commit introduces a comprehensive integration with KOReader Sync Server to enable two-way synchronization of reading progress. The core logic is encapsulated in a new `KoreaderSyncService`, which handles authentication, registration, and progress pushing/pulling based on user-defined strategies (LATEST, KOSYNC, SUWAYOMI). Key changes include: - A new GraphQL API is added to manage the KOReader Sync connection: - `connectKoSyncAccount` mutation provides a simplified flow that attempts to log in and, if the user doesn't exist, automatically registers them. - `logoutKoSyncAccount` mutation to clear credentials. - `koSyncStatus` query to check the current connection status. - Reading progress is now synchronized at key points: - The `fetchChapterPages` mutation pulls the latest progress from the sync server before loading the reader. It respects the configured sync strategy and updates the local database if necessary. - The `updateChapters` and other progress-updating methods now push changes to the sync server automatically. - OPDS chapter entries also pull the latest progress, ensuring clients receive up-to-date reading status. - Supporting backend changes have been made: - The `Chapter` table is extended with a `koreader_hash` column to uniquely identify documents. A database migration is included. - New configuration options are added to `server.conf` to manage the feature (enable, server URL, credentials, strategy, etc.). * perf(opds): defer KOReader sync to improve chapter feed performance Removes the KOReader Sync progress-pulling logic from the `createChapterListEntry` function. The previous implementation triggered a network request to the sync server for every single chapter when generating a list, leading to severe performance issues and slow load times on feeds with many entries. This change reverts to the more performant approach of always linking to the chapter's metadata feed. The expensive sync operation will be handled within the metadata entry generation instead, ensuring it's only triggered on-demand for a single chapter. This restores the responsiveness of browsing chapter feeds. * refactor(koreader): Use enums for sync settings and correct OPDS logic Refactor Koreader Sync settings to use enums instead of raw strings for `checksumMethod` and `strategy`. This improves type safety, prevents typos, and makes the configuration handling more robust. The changes include: - Introducing `KoreaderSyncChecksumMethod` and `KoreaderSyncStrategy` enums. - Updating `ServerConfig`, GraphQL types, and backup models to use these new enums. - Refactoring `KoreaderSyncService` to work with the enum types. Additionally, this commit fixes an issue in `OpdsEntryBuilder` where the logic for determining which sync progress to use (local vs. remote) was duplicated. The builder now correctly delegates this decision to `KoreaderSyncService.pullProgress`, which already contains the necessary strategy logic. This centralizes the logic and ensures consistent behavior. * refactor(koreader): Improve config handling and remove redundant update This commit combines several refactoring and cleanup tasks: - **Koreader Sync:** The sync service is updated to use the modern `serverConfig` provider instead of the legacy `GlobalConfigManager`. This aligns it with the current configuration management approach in the project. - **Download Provider:** A redundant `pageCount` database update is removed from `ChaptersFilesProvider`. This operation was unnecessary because the `getChapterDownloadReady` function, which is called earlier in the download process, already verifies and corrects the page count. This change eliminates a superfluous database write and fixes a related import issue. * feat(sync/koreader)!: enhance sync strategy and add progress tolerance This commit overhauls the KOReader synchronization feature to provide more granular control and robustness. The simple on/off toggle has been replaced with a more flexible strategy-based system. Key changes include: - Replaced `koreaderSyncEnabled` with a more powerful `koreaderSyncStrategy` enum. - Introduced new sync strategies: `PROMPT`, `SILENT`, `SEND`, `RECEIVE`, and `DISABLE`, allowing for fine-grained control over the sync direction and conflict resolution. - Added a `koreaderSyncPercentageTolerance` setting. This prevents unnecessary sync updates for minor progress differences between Suwayomi and KOReader. - Refactored the `KoreaderSyncService` to implement the new strategies and use the configurable tolerance. - Updated GraphQL schemas, mutations, and server configuration to remove the old setting and incorporate the new ones. - Adjusted the backup and restore process to correctly handle the new configuration parameters. - Modified API endpoints and internal logic to check and apply remote progress based on the selected strategy. BREAKING CHANGE: The `koreaderSyncEnabled` setting is removed and replaced by a more granular `koreaderSyncStrategy`. The enum values for the strategy have been completely changed, making previous configurations incompatible. * fix: remove unused imports * feat(opds, sync): enhance Koreader sync and OPDS conflict handling This commit introduces significant improvements to the Koreader synchronization feature, focusing on providing a better user experience for handling progress conflicts in both OPDS and GraphQL clients. Key changes include: - **OPDS Conflict Resolution:** When a reading progress conflict is detected, the OPDS feed for a chapter now provides two distinct "Read Online" links: one to continue from the local progress and another to continue from the synced progress from the remote device (e.g., "Continue Reading Online (Synced from KOReader)"). This empowers users to choose which progress to follow. - **GraphQL Sync Conflict Information:** The `fetchChapterPages` GraphQL mutation now includes a `syncConflict` field in its payload. This field provides the remote device name and page number, allowing GraphQL clients to implement a user-facing prompt to resolve sync conflicts. - **Improved Sync Strategy Handling:** - The `connectKoSyncAccount` mutation no longer unconditionally sets the sync strategy to `PROMPT`. It now respects the user's existing setting, preventing accidental configuration changes upon re-login. - The default `koreaderSyncStrategy` in the configuration is changed to `DISABLED`, providing a safer and more intuitive default for new users. - **Refinements & Fixes:** - The fallback for the remote device name is now centralized within the KoreaderSyncService, defaulting to "KOReader" if not provided. - Renamed `KoreaderSyncStrategy.DISABLE` to `DISABLED` for consistency. - Updated i18n strings for OPDS links to be more descriptive and user-friendly. * refactor(kosync): rename stream page link titles for consistency * refactor(kosync): return SettingsType in auth mutation payloads The `connectKoSyncAccount` and `logoutKoSyncAccount` mutations modify server settings (username and userkey) but did not previously return the updated configuration. This forced client applications to manually refetch settings to avoid a stale cache. This change modifies the payloads for both mutations to include the full `SettingsType`. By returning the updated settings directly, GraphQL clients like Apollo Client can automatically update their cache, simplifying client-side state management and ensuring the UI always reflects the current server configuration. Additionally, `clientMutationId` has been added to `KoSyncConnectPayload` for consistency with GraphQL practices, aligning it with the logout mutation. Refs: #1560 * refactor(kosync): replace KoSyncConnectPayload with ConnectResult in connect method * fix(kosync): add koreaderSyncPercentageTolerance default setting --- .../moko-resources/values/base/strings.xml | 9 +- .../graphql/mutations/ChapterMutation.kt | 52 ++- .../graphql/mutations/KoreaderSyncMutation.kt | 43 ++ .../graphql/mutations/SettingsMutation.kt | 9 + .../graphql/queries/KoreaderSyncQuery.kt | 13 + .../graphql/server/TachideskGraphQLSchema.kt | 4 + .../tachidesk/graphql/types/ChapterType.kt | 5 + .../graphql/types/KoreaderSyncTypes.kt | 33 ++ .../tachidesk/graphql/types/SettingsType.kt | 33 ++ .../manga/controller/MangaController.kt | 44 +- .../suwayomi/tachidesk/manga/impl/Chapter.kt | 57 ++- .../impl/backup/proto/ProtoBackupExport.kt | 8 + .../proto/models/BackupServerSettings.kt | 10 + .../fileProvider/ChaptersFilesProvider.kt | 24 +- .../fileProvider/impl/ArchiveProvider.kt | 10 + .../fileProvider/impl/FolderProvider.kt | 10 + .../manga/impl/sync/KoreaderSyncService.kt | 425 ++++++++++++++++++ .../manga/impl/util/KoreaderHelper.kt | 48 ++ .../manga/model/table/ChapterTable.kt | 2 + .../tachidesk/opds/impl/OpdsEntryBuilder.kt | 98 +++- .../suwayomi/tachidesk/server/ServerConfig.kt | 11 + .../M0051_AddKoreaderHashToChapterTable.kt | 19 + .../src/main/resources/server-reference.conf | 9 + 23 files changed, 926 insertions(+), 50 deletions(-) create mode 100644 server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/KoreaderSyncMutation.kt create mode 100644 server/src/main/kotlin/suwayomi/tachidesk/graphql/queries/KoreaderSyncQuery.kt create mode 100644 server/src/main/kotlin/suwayomi/tachidesk/graphql/types/KoreaderSyncTypes.kt create mode 100644 server/src/main/kotlin/suwayomi/tachidesk/manga/impl/sync/KoreaderSyncService.kt create mode 100644 server/src/main/kotlin/suwayomi/tachidesk/manga/impl/util/KoreaderHelper.kt create mode 100644 server/src/main/kotlin/suwayomi/tachidesk/server/database/migration/M0051_AddKoreaderHashToChapterTable.kt diff --git a/server/i18n/src/commonMain/moko-resources/values/base/strings.xml b/server/i18n/src/commonMain/moko-resources/values/base/strings.xml index 10703314..b55c3da5 100644 --- a/server/i18n/src/commonMain/moko-resources/values/base/strings.xml +++ b/server/i18n/src/commonMain/moko-resources/values/base/strings.xml @@ -95,7 +95,12 @@ Next Page Current Feed View on Web - View Pages (Streaming) + Read Online + Continue Reading Online + Read Online (Local Progress) + Continue Reading Online (Local Progress) + Read Online (Synced from %1$s) + Continue Reading Online (Synced from %1$s) Download CBZ Chapter Cover View Chapter Details & Get Pages @@ -106,7 +111,7 @@ ⭕ ⬇️ - %1$s | %2$s + Series: %1$s | %2$s | By %1$s | Progress: %1$d of %2$d diff --git a/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/ChapterMutation.kt b/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/ChapterMutation.kt index 50824807..a73fd8b5 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/ChapterMutation.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/ChapterMutation.kt @@ -1,6 +1,8 @@ package suwayomi.tachidesk.graphql.mutations import graphql.execution.DataFetcherResult +import kotlinx.coroutines.GlobalScope +import kotlinx.coroutines.launch import org.jetbrains.exposed.dao.id.EntityID import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq import org.jetbrains.exposed.sql.and @@ -8,11 +10,14 @@ import org.jetbrains.exposed.sql.deleteWhere import org.jetbrains.exposed.sql.selectAll import org.jetbrains.exposed.sql.statements.BatchUpdateStatement import org.jetbrains.exposed.sql.transactions.transaction +import org.jetbrains.exposed.sql.update import suwayomi.tachidesk.graphql.asDataFetcherResult import suwayomi.tachidesk.graphql.types.ChapterMetaType import suwayomi.tachidesk.graphql.types.ChapterType +import suwayomi.tachidesk.graphql.types.SyncConflictInfoType import suwayomi.tachidesk.manga.impl.Chapter import suwayomi.tachidesk.manga.impl.chapter.getChapterDownloadReadyById +import suwayomi.tachidesk.manga.impl.sync.KoreaderSyncService import suwayomi.tachidesk.manga.model.table.ChapterMetaTable import suwayomi.tachidesk.manga.model.table.ChapterTable import suwayomi.tachidesk.server.JavalinSetup.future @@ -68,8 +73,10 @@ class ChapterMutation { ChapterTable .select(ChapterTable.id, ChapterTable.pageCount) .where { ChapterTable.id inList ids } - .groupBy { it[ChapterTable.id].value } - .mapValues { it.value.firstOrNull()?.let { it[ChapterTable.pageCount] } } + .associateBy( + { it[ChapterTable.id].value }, + { it[ChapterTable.pageCount] }, + ) } else { emptyMap() } @@ -94,6 +101,15 @@ class ChapterMutation { } } } + + // Sync with KoreaderSync when progress is updated + if (patch.lastPageRead != null || patch.isRead == true) { + GlobalScope.launch { + ids.forEach { chapterId -> + KoreaderSyncService.pushProgress(chapterId) + } + } + } } fun updateChapter(input: UpdateChapterInput): DataFetcherResult = @@ -241,6 +257,7 @@ class ChapterMutation { val clientMutationId: String?, val pages: List, val chapter: ChapterType, + val syncConflict: SyncConflictInfoType?, ) fun fetchChapterPages(input: FetchChapterPagesInput): CompletableFuture> { @@ -249,7 +266,35 @@ class ChapterMutation { return future { asDataFetcherResult { - val chapter = getChapterDownloadReadyById(chapterId) + var chapter = getChapterDownloadReadyById(chapterId) + val syncResult = KoreaderSyncService.checkAndPullProgress(chapter.id) + var syncConflictInfo: SyncConflictInfoType? = null + + if (syncResult != null) { + if (syncResult.isConflict) { + syncConflictInfo = + SyncConflictInfoType( + deviceName = syncResult.device, + remotePage = syncResult.pageRead, + ) + } + + if (syncResult.shouldUpdate) { + // Update DB for SILENT and RECEIVE + transaction { + ChapterTable.update({ ChapterTable.id eq chapter.id }) { + it[lastPageRead] = syncResult.pageRead + it[lastReadAt] = syncResult.timestamp + } + } + } + // For PROMPT, SILENT, and RECEIVE, return the remote progress + chapter = + chapter.copy( + lastPageRead = if (syncResult.shouldUpdate) syncResult.pageRead else chapter.lastPageRead, + lastReadAt = if (syncResult.shouldUpdate) syncResult.timestamp else chapter.lastReadAt, + ) + } val params = buildString { @@ -273,6 +318,7 @@ class ChapterMutation { "/api/v1/manga/${chapter.mangaId}/chapter/${chapter.index}/page/${index}$params" }, chapter = ChapterType(chapter), + syncConflict = syncConflictInfo, ) } } diff --git a/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/KoreaderSyncMutation.kt b/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/KoreaderSyncMutation.kt new file mode 100644 index 00000000..feafc6aa --- /dev/null +++ b/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/KoreaderSyncMutation.kt @@ -0,0 +1,43 @@ +package suwayomi.tachidesk.graphql.mutations + +import suwayomi.tachidesk.graphql.types.KoSyncConnectPayload +import suwayomi.tachidesk.graphql.types.LogoutKoSyncAccountPayload +import suwayomi.tachidesk.graphql.types.SettingsType +import suwayomi.tachidesk.manga.impl.sync.KoreaderSyncService +import suwayomi.tachidesk.server.JavalinSetup.future +import java.util.concurrent.CompletableFuture + +class KoreaderSyncMutation { + data class ConnectKoSyncAccountInput( + val clientMutationId: String? = null, + val username: String, + val password: String, + ) + + fun connectKoSyncAccount(input: ConnectKoSyncAccountInput): CompletableFuture = + future { + val result = KoreaderSyncService.connect(input.username, input.password) + + KoSyncConnectPayload( + clientMutationId = input.clientMutationId, + success = result.success, + message = result.message, + username = result.username, + settings = SettingsType(), + ) + } + + data class LogoutKoSyncAccountInput( + val clientMutationId: String? = null, + ) + + fun logoutKoSyncAccount(input: LogoutKoSyncAccountInput): CompletableFuture = + future { + KoreaderSyncService.logout() + LogoutKoSyncAccountPayload( + clientMutationId = input.clientMutationId, + success = true, + settings = SettingsType(), + ) + } +} diff --git a/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/SettingsMutation.kt b/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/SettingsMutation.kt index 4cc0bc1c..09e5009c 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/SettingsMutation.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/graphql/mutations/SettingsMutation.kt @@ -215,6 +215,15 @@ class SettingsMutation { updateSetting(settings.opdsShowOnlyUnreadChapters, serverConfig.opdsShowOnlyUnreadChapters) updateSetting(settings.opdsShowOnlyDownloadedChapters, serverConfig.opdsShowOnlyDownloadedChapters) updateSetting(settings.opdsChapterSortOrder, serverConfig.opdsChapterSortOrder) + + // koreader sync + updateSetting(settings.koreaderSyncServerUrl, serverConfig.koreaderSyncServerUrl) + updateSetting(settings.koreaderSyncUsername, serverConfig.koreaderSyncUsername) + updateSetting(settings.koreaderSyncUserkey, serverConfig.koreaderSyncUserkey) + updateSetting(settings.koreaderSyncDeviceId, serverConfig.koreaderSyncDeviceId) + updateSetting(settings.koreaderSyncChecksumMethod, serverConfig.koreaderSyncChecksumMethod) + updateSetting(settings.koreaderSyncStrategy, serverConfig.koreaderSyncStrategy) + updateSetting(settings.koreaderSyncPercentageTolerance, serverConfig.koreaderSyncPercentageTolerance) } fun setSettings(input: SetSettingsInput): SetSettingsPayload { diff --git a/server/src/main/kotlin/suwayomi/tachidesk/graphql/queries/KoreaderSyncQuery.kt b/server/src/main/kotlin/suwayomi/tachidesk/graphql/queries/KoreaderSyncQuery.kt new file mode 100644 index 00000000..dd20e406 --- /dev/null +++ b/server/src/main/kotlin/suwayomi/tachidesk/graphql/queries/KoreaderSyncQuery.kt @@ -0,0 +1,13 @@ +package suwayomi.tachidesk.graphql.queries + +import suwayomi.tachidesk.graphql.types.KoSyncStatusPayload +import suwayomi.tachidesk.manga.impl.sync.KoreaderSyncService +import suwayomi.tachidesk.server.JavalinSetup.future +import java.util.concurrent.CompletableFuture + +class KoreaderSyncQuery { + fun koSyncStatus(): CompletableFuture = + future { + KoreaderSyncService.getStatus() + } +} diff --git a/server/src/main/kotlin/suwayomi/tachidesk/graphql/server/TachideskGraphQLSchema.kt b/server/src/main/kotlin/suwayomi/tachidesk/graphql/server/TachideskGraphQLSchema.kt index ae2ce146..9af55a9b 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/graphql/server/TachideskGraphQLSchema.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/graphql/server/TachideskGraphQLSchema.kt @@ -20,6 +20,7 @@ import suwayomi.tachidesk.graphql.mutations.DownloadMutation import suwayomi.tachidesk.graphql.mutations.ExtensionMutation import suwayomi.tachidesk.graphql.mutations.ImageMutation import suwayomi.tachidesk.graphql.mutations.InfoMutation +import suwayomi.tachidesk.graphql.mutations.KoreaderSyncMutation import suwayomi.tachidesk.graphql.mutations.MangaMutation import suwayomi.tachidesk.graphql.mutations.MetaMutation import suwayomi.tachidesk.graphql.mutations.SettingsMutation @@ -32,6 +33,7 @@ import suwayomi.tachidesk.graphql.queries.ChapterQuery import suwayomi.tachidesk.graphql.queries.DownloadQuery import suwayomi.tachidesk.graphql.queries.ExtensionQuery import suwayomi.tachidesk.graphql.queries.InfoQuery +import suwayomi.tachidesk.graphql.queries.KoreaderSyncQuery import suwayomi.tachidesk.graphql.queries.MangaQuery import suwayomi.tachidesk.graphql.queries.MetaQuery import suwayomi.tachidesk.graphql.queries.SettingsQuery @@ -74,6 +76,7 @@ val schema = TopLevelObject(DownloadQuery()), TopLevelObject(ExtensionQuery()), TopLevelObject(InfoQuery()), + TopLevelObject(KoreaderSyncQuery()), TopLevelObject(MangaQuery()), TopLevelObject(MetaQuery()), TopLevelObject(SettingsQuery()), @@ -90,6 +93,7 @@ val schema = TopLevelObject(ExtensionMutation()), TopLevelObject(ImageMutation()), TopLevelObject(InfoMutation()), + TopLevelObject(KoreaderSyncMutation()), TopLevelObject(MangaMutation()), TopLevelObject(MetaMutation()), TopLevelObject(SettingsMutation()), diff --git a/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/ChapterType.kt b/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/ChapterType.kt index e12d4b7c..09c30f90 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/ChapterType.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/ChapterType.kt @@ -19,6 +19,11 @@ import suwayomi.tachidesk.manga.model.dataclass.ChapterDataClass import suwayomi.tachidesk.manga.model.table.ChapterTable import java.util.concurrent.CompletableFuture +data class SyncConflictInfoType( + val deviceName: String, + val remotePage: Int, +) + class ChapterType( val id: Int, val url: String, diff --git a/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/KoreaderSyncTypes.kt b/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/KoreaderSyncTypes.kt new file mode 100644 index 00000000..26343f7f --- /dev/null +++ b/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/KoreaderSyncTypes.kt @@ -0,0 +1,33 @@ +package suwayomi.tachidesk.graphql.types + +enum class KoreaderSyncChecksumMethod { + BINARY, + FILENAME, +} + +enum class KoreaderSyncStrategy { + PROMPT, // Ask on conflict + SILENT, // Always use latest + SEND, // Send changes only + RECEIVE, // Receive changes only + DISABLED, +} + +data class KoSyncStatusPayload( + val isLoggedIn: Boolean, + val username: String?, +) + +data class KoSyncConnectPayload( + val clientMutationId: String?, + val success: Boolean, + val message: String?, + val username: String?, + val settings: SettingsType, +) + +data class LogoutKoSyncAccountPayload( + val clientMutationId: String?, + val success: Boolean, + val settings: SettingsType, +) diff --git a/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/SettingsType.kt b/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/SettingsType.kt index c5f66c38..2bac6679 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/SettingsType.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/graphql/types/SettingsType.kt @@ -112,6 +112,15 @@ interface Settings : Node { val opdsShowOnlyUnreadChapters: Boolean? val opdsShowOnlyDownloadedChapters: Boolean? val opdsChapterSortOrder: SortOrder? + + // koreader sync + val koreaderSyncServerUrl: String? + val koreaderSyncUsername: String? + val koreaderSyncUserkey: String? + val koreaderSyncDeviceId: String? + val koreaderSyncChecksumMethod: KoreaderSyncChecksumMethod? + val koreaderSyncStrategy: KoreaderSyncStrategy? + val koreaderSyncPercentageTolerance: Double? } interface SettingsDownloadConversion { @@ -206,6 +215,14 @@ data class PartialSettingsType( override val opdsShowOnlyUnreadChapters: Boolean?, override val opdsShowOnlyDownloadedChapters: Boolean?, override val opdsChapterSortOrder: SortOrder?, + // koreader sync + override val koreaderSyncServerUrl: String?, + override val koreaderSyncUsername: String?, + override val koreaderSyncUserkey: String?, + override val koreaderSyncDeviceId: String?, + override val koreaderSyncChecksumMethod: KoreaderSyncChecksumMethod?, + override val koreaderSyncStrategy: KoreaderSyncStrategy?, + override val koreaderSyncPercentageTolerance: Double?, ) : Settings class SettingsType( @@ -288,6 +305,14 @@ class SettingsType( override val opdsShowOnlyUnreadChapters: Boolean, override val opdsShowOnlyDownloadedChapters: Boolean, override val opdsChapterSortOrder: SortOrder, + // koreader sync + override val koreaderSyncServerUrl: String, + override val koreaderSyncUsername: String, + override val koreaderSyncUserkey: String, + override val koreaderSyncDeviceId: String, + override val koreaderSyncChecksumMethod: KoreaderSyncChecksumMethod, + override val koreaderSyncStrategy: KoreaderSyncStrategy, + override val koreaderSyncPercentageTolerance: Double, ) : Settings { constructor(config: ServerConfig = serverConfig) : this( config.ip.value, @@ -367,5 +392,13 @@ class SettingsType( config.opdsShowOnlyUnreadChapters.value, config.opdsShowOnlyDownloadedChapters.value, config.opdsChapterSortOrder.value, + // koreader sync + config.koreaderSyncServerUrl.value, + config.koreaderSyncUsername.value, + config.koreaderSyncUserkey.value, + config.koreaderSyncDeviceId.value, + config.koreaderSyncChecksumMethod.value, + config.koreaderSyncStrategy.value, + config.koreaderSyncPercentageTolerance.value, ) } diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/controller/MangaController.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/controller/MangaController.kt index c875d15b..af4a5c50 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/manga/controller/MangaController.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/controller/MangaController.kt @@ -9,7 +9,11 @@ package suwayomi.tachidesk.manga.controller import io.javalin.http.HandlerType import io.javalin.http.HttpStatus +import kotlinx.coroutines.GlobalScope +import kotlinx.coroutines.launch import kotlinx.serialization.json.Json +import org.jetbrains.exposed.sql.transactions.transaction +import org.jetbrains.exposed.sql.update import suwayomi.tachidesk.manga.impl.CategoryManga import suwayomi.tachidesk.manga.impl.Chapter import suwayomi.tachidesk.manga.impl.ChapterDownloadHelper @@ -17,9 +21,11 @@ import suwayomi.tachidesk.manga.impl.Library import suwayomi.tachidesk.manga.impl.Manga import suwayomi.tachidesk.manga.impl.Page import suwayomi.tachidesk.manga.impl.chapter.getChapterDownloadReadyByIndex +import suwayomi.tachidesk.manga.impl.sync.KoreaderSyncService import suwayomi.tachidesk.manga.model.dataclass.CategoryDataClass import suwayomi.tachidesk.manga.model.dataclass.ChapterDataClass import suwayomi.tachidesk.manga.model.dataclass.MangaDataClass +import suwayomi.tachidesk.manga.model.table.ChapterTable import suwayomi.tachidesk.server.JavalinSetup.future import suwayomi.tachidesk.server.util.formParam import suwayomi.tachidesk.server.util.handler @@ -315,8 +321,29 @@ object MangaController { }, behaviorOf = { ctx, mangaId, chapterIndex -> ctx.future { - future { getChapterDownloadReadyByIndex(chapterIndex, mangaId) } - .thenApply { ctx.json(it) } + future { + var chapter = getChapterDownloadReadyByIndex(chapterIndex, mangaId) + val syncResult = KoreaderSyncService.checkAndPullProgress(chapter.id) + + if (syncResult != null) { + if (syncResult.shouldUpdate) { + // Update DB for SILENT and RECEIVE + transaction { + ChapterTable.update({ ChapterTable.id eq chapter.id }) { + it[lastPageRead] = syncResult.pageRead + it[lastReadAt] = syncResult.timestamp + } + } + } + // For PROMPT, SILENT, and RECEIVE, return the remote progress + chapter = + chapter.copy( + lastPageRead = syncResult.pageRead, + lastReadAt = syncResult.timestamp, + ) + } + chapter + }.thenApply { ctx.json(it) } } }, withResults = { @@ -341,7 +368,12 @@ object MangaController { } }, behaviorOf = { ctx, mangaId, chapterIndex, read, bookmarked, markPrevRead, lastPageRead -> - Chapter.modifyChapter(mangaId, chapterIndex, read, bookmarked, markPrevRead, lastPageRead) + val chapterId = Chapter.modifyChapter(mangaId, chapterIndex, read, bookmarked, markPrevRead, lastPageRead) + + // Sync with KoreaderSync when progress is updated + if (lastPageRead != null || read == true) { + GlobalScope.launch { KoreaderSyncService.pushProgress(chapterId) } + } ctx.status(200) }, @@ -422,7 +454,11 @@ object MangaController { ctx.result(it.first) if (updateProgress == true) { - Chapter.updateChapterProgress(mangaId, chapterIndex, pageNo = index) + val chapterId = Chapter.updateChapterProgress(mangaId, chapterIndex, pageNo = index) + // Sync progress with KoreaderSync if chapter update was successful + if (chapterId != -1) { + GlobalScope.launch { KoreaderSyncService.pushProgress(chapterId) } + } } } } diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/Chapter.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/Chapter.kt index 55017f28..d3fceb71 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/Chapter.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/Chapter.kt @@ -431,33 +431,46 @@ object Chapter { isBookmarked: Boolean?, markPrevRead: Boolean?, lastPageRead: Int?, - ) { - transaction { - if (listOf(isRead, isBookmarked, lastPageRead).any { it != null }) { - ChapterTable.update({ (ChapterTable.manga eq mangaId) and (ChapterTable.sourceOrder eq chapterIndex) }) { update -> - isRead?.also { - update[ChapterTable.isRead] = it - } - isBookmarked?.also { - update[ChapterTable.isBookmarked] = it - } - lastPageRead?.also { - update[ChapterTable.lastPageRead] = it - update[lastReadAt] = Instant.now().epochSecond - } - } - } + ): Int { + val chapterId = + transaction { + val chapter = + ChapterTable + .selectAll() + .where { (ChapterTable.manga eq mangaId) and (ChapterTable.sourceOrder eq chapterIndex) } + .first() - markPrevRead?.let { - ChapterTable.update({ (ChapterTable.manga eq mangaId) and (ChapterTable.sourceOrder less chapterIndex) }) { - it[ChapterTable.isRead] = markPrevRead + val chapterIdValue = chapter[ChapterTable.id].value + + if (listOf(isRead, isBookmarked, lastPageRead).any { it != null }) { + ChapterTable.update({ (ChapterTable.manga eq mangaId) and (ChapterTable.sourceOrder eq chapterIndex) }) { update -> + isRead?.also { + update[ChapterTable.isRead] = it + } + isBookmarked?.also { + update[ChapterTable.isBookmarked] = it + } + lastPageRead?.also { + update[ChapterTable.lastPageRead] = it + update[lastReadAt] = Instant.now().epochSecond + } + } } + + markPrevRead?.let { + ChapterTable.update({ (ChapterTable.manga eq mangaId) and (ChapterTable.sourceOrder less chapterIndex) }) { + it[ChapterTable.isRead] = markPrevRead + } + } + + chapterIdValue } - } if (isRead == true || markPrevRead == true) { Track.asyncTrackChapter(setOf(mangaId)) } + + return chapterId } @Serializable @@ -733,7 +746,7 @@ object Chapter { mangaId: Int, chapterIndex: Int, pageNo: Int, - ) { + ): Int { val chapterData = transaction { ChapterTable @@ -756,5 +769,7 @@ object Chapter { isBookmarked = null, markPrevRead = null, ) + + return chapterData.id } } diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/backup/proto/ProtoBackupExport.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/backup/proto/ProtoBackupExport.kt index d7846d1e..c2003ac1 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/backup/proto/ProtoBackupExport.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/backup/proto/ProtoBackupExport.kt @@ -463,6 +463,14 @@ object ProtoBackupExport : ProtoBackupBase() { opdsShowOnlyUnreadChapters = serverConfig.opdsShowOnlyUnreadChapters.value, opdsShowOnlyDownloadedChapters = serverConfig.opdsShowOnlyDownloadedChapters.value, opdsChapterSortOrder = serverConfig.opdsChapterSortOrder.value, + // koreader sync + koreaderSyncServerUrl = serverConfig.koreaderSyncServerUrl.value, + koreaderSyncUsername = serverConfig.koreaderSyncUsername.value, + koreaderSyncUserkey = serverConfig.koreaderSyncUserkey.value, + koreaderSyncDeviceId = serverConfig.koreaderSyncDeviceId.value, + koreaderSyncChecksumMethod = serverConfig.koreaderSyncChecksumMethod.value, + koreaderSyncStrategy = serverConfig.koreaderSyncStrategy.value, + koreaderSyncPercentageTolerance = serverConfig.koreaderSyncPercentageTolerance.value, ) } } diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/backup/proto/models/BackupServerSettings.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/backup/proto/models/BackupServerSettings.kt index dae300d2..6e9c1510 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/backup/proto/models/BackupServerSettings.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/backup/proto/models/BackupServerSettings.kt @@ -4,6 +4,8 @@ import kotlinx.serialization.Serializable import kotlinx.serialization.protobuf.ProtoNumber import org.jetbrains.exposed.sql.SortOrder import suwayomi.tachidesk.graphql.types.AuthMode +import suwayomi.tachidesk.graphql.types.KoreaderSyncChecksumMethod +import suwayomi.tachidesk.graphql.types.KoreaderSyncStrategy import suwayomi.tachidesk.graphql.types.Settings import suwayomi.tachidesk.graphql.types.SettingsDownloadConversion import suwayomi.tachidesk.graphql.types.WebUIChannel @@ -84,6 +86,14 @@ data class BackupServerSettings( @ProtoNumber(53) override var opdsShowOnlyUnreadChapters: Boolean, @ProtoNumber(54) override var opdsShowOnlyDownloadedChapters: Boolean, @ProtoNumber(55) override var opdsChapterSortOrder: SortOrder, + // koreader sync + @ProtoNumber(59) override var koreaderSyncServerUrl: String, + @ProtoNumber(60) override var koreaderSyncUsername: String, + @ProtoNumber(61) override var koreaderSyncUserkey: String, + @ProtoNumber(62) override var koreaderSyncDeviceId: String, + @ProtoNumber(63) override var koreaderSyncChecksumMethod: KoreaderSyncChecksumMethod, + @ProtoNumber(64) override var koreaderSyncStrategy: KoreaderSyncStrategy, + @ProtoNumber(65) override var koreaderSyncPercentageTolerance: Double, ) : Settings { @Serializable class BackupSettingsDownloadConversionType( diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/ChaptersFilesProvider.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/ChaptersFilesProvider.kt index 939b7a7d..fa48312d 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/ChaptersFilesProvider.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/ChaptersFilesProvider.kt @@ -13,11 +13,14 @@ import libcore.net.MimeUtils import org.apache.commons.compress.archivers.zip.ZipArchiveEntry import org.jetbrains.exposed.sql.selectAll import org.jetbrains.exposed.sql.transactions.transaction +import org.jetbrains.exposed.sql.update import suwayomi.tachidesk.manga.impl.Page import suwayomi.tachidesk.manga.impl.chapter.getChapterDownloadReady import suwayomi.tachidesk.manga.impl.download.model.DownloadChapter +import suwayomi.tachidesk.manga.impl.util.KoreaderHelper import suwayomi.tachidesk.manga.impl.util.createComicInfoFile import suwayomi.tachidesk.manga.impl.util.getChapterCachePath +import suwayomi.tachidesk.manga.impl.util.getChapterCbzPath import suwayomi.tachidesk.manga.impl.util.getChapterDownloadPath import suwayomi.tachidesk.manga.impl.util.storage.ImageResponse import suwayomi.tachidesk.manga.model.table.ChapterTable @@ -43,20 +46,20 @@ sealed class FileType { fun getName(): String = when (this) { - is FileType.RegularFile -> { + is RegularFile -> { this.file.name } - is FileType.ZipFile -> { + is ZipFile -> { this.entry.name } } fun getExtension(): String = when (this) { - is FileType.RegularFile -> { + is RegularFile -> { this.file.extension } - is FileType.ZipFile -> { + is ZipFile -> { this.entry.name.substringAfterLast(".") } } @@ -189,6 +192,19 @@ abstract class ChaptersFilesProvider( handleSuccessfulDownload() + // Calculate and save Koreader hash for CBZ files + val chapterFile = File(getChapterCbzPath(mangaId, chapterId)) + if (chapterFile.exists()) { + val koreaderHash = KoreaderHelper.hashContents(chapterFile) + if (koreaderHash != null) { + transaction { + ChapterTable.update({ ChapterTable.id eq chapterId }) { + it[ChapterTable.koreaderHash] = koreaderHash + } + } + } + } + File(cacheChapterDir).deleteRecursively() return true diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/impl/ArchiveProvider.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/impl/ArchiveProvider.kt index 4caa9e87..f3ca3549 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/impl/ArchiveProvider.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/impl/ArchiveProvider.kt @@ -6,6 +6,8 @@ import org.apache.commons.compress.archivers.zip.ZipArchiveEntry import org.apache.commons.compress.archivers.zip.ZipArchiveInputStream import org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream import org.apache.commons.compress.archivers.zip.ZipFile +import org.jetbrains.exposed.sql.transactions.transaction +import org.jetbrains.exposed.sql.update import suwayomi.tachidesk.manga.impl.download.fileProvider.ChaptersFilesProvider import suwayomi.tachidesk.manga.impl.download.fileProvider.FileType import suwayomi.tachidesk.manga.impl.util.getChapterCachePath @@ -13,6 +15,7 @@ import suwayomi.tachidesk.manga.impl.util.getChapterCbzPath import suwayomi.tachidesk.manga.impl.util.getChapterDownloadPath import suwayomi.tachidesk.manga.impl.util.getMangaDownloadDir import suwayomi.tachidesk.manga.impl.util.storage.FileDeletionHelper +import suwayomi.tachidesk.manga.model.table.ChapterTable import suwayomi.tachidesk.server.ApplicationDirs import uy.kohesive.injekt.injectLazy import java.io.File @@ -85,6 +88,13 @@ class ArchiveProvider( } val cbzDeleted = cbzFile.delete() + if (cbzDeleted) { + transaction { + ChapterTable.update({ ChapterTable.id eq chapterId }) { + it[koreaderHash] = null + } + } + } FileDeletionHelper.cleanupParentFoldersFor(cbzFile, applicationDirs.mangaDownloadsRoot) return cbzDeleted } diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/impl/FolderProvider.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/impl/FolderProvider.kt index 5f84c7b7..c11296fd 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/impl/FolderProvider.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/download/fileProvider/impl/FolderProvider.kt @@ -1,10 +1,13 @@ package suwayomi.tachidesk.manga.impl.download.fileProvider.impl +import org.jetbrains.exposed.sql.transactions.transaction +import org.jetbrains.exposed.sql.update import suwayomi.tachidesk.manga.impl.download.fileProvider.ChaptersFilesProvider import suwayomi.tachidesk.manga.impl.download.fileProvider.FileType.RegularFile import suwayomi.tachidesk.manga.impl.util.getChapterCachePath import suwayomi.tachidesk.manga.impl.util.getChapterDownloadPath import suwayomi.tachidesk.manga.impl.util.storage.FileDeletionHelper +import suwayomi.tachidesk.manga.model.table.ChapterTable import suwayomi.tachidesk.server.ApplicationDirs import uy.kohesive.injekt.injectLazy import java.io.BufferedOutputStream @@ -61,6 +64,13 @@ class FolderProvider( } val chapterDirDeleted = chapterDir.deleteRecursively() + if (chapterDirDeleted) { + transaction { + ChapterTable.update({ ChapterTable.id eq chapterId }) { + it[koreaderHash] = null + } + } + } FileDeletionHelper.cleanupParentFoldersFor(chapterDir, applicationDirs.mangaDownloadsRoot) return chapterDirDeleted } diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/sync/KoreaderSyncService.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/sync/KoreaderSyncService.kt new file mode 100644 index 00000000..3275980c --- /dev/null +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/sync/KoreaderSyncService.kt @@ -0,0 +1,425 @@ +package suwayomi.tachidesk.manga.impl.sync + +import eu.kanade.tachiyomi.network.NetworkHelper +import eu.kanade.tachiyomi.network.await +import eu.kanade.tachiyomi.util.lang.Hash +import io.github.oshai.kotlinlogging.KotlinLogging +import io.javalin.json.JsonMapper +import kotlinx.serialization.Serializable +import kotlinx.serialization.json.Json +import kotlinx.serialization.json.buildJsonObject +import kotlinx.serialization.json.put +import okhttp3.MediaType.Companion.toMediaType +import okhttp3.Request +import okhttp3.RequestBody.Companion.toRequestBody +import org.jetbrains.exposed.sql.transactions.transaction +import org.jetbrains.exposed.sql.update +import suwayomi.tachidesk.graphql.types.KoSyncStatusPayload +import suwayomi.tachidesk.graphql.types.KoreaderSyncChecksumMethod +import suwayomi.tachidesk.graphql.types.KoreaderSyncStrategy +import suwayomi.tachidesk.manga.impl.util.KoreaderHelper +import suwayomi.tachidesk.manga.impl.util.getChapterCbzPath +import suwayomi.tachidesk.manga.model.table.ChapterTable +import suwayomi.tachidesk.manga.model.table.MangaTable +import suwayomi.tachidesk.server.serverConfig +import uy.kohesive.injekt.injectLazy +import java.io.File +import java.util.UUID +import kotlin.math.abs + +object KoreaderSyncService { + private val logger = KotlinLogging.logger {} + private val network: NetworkHelper by injectLazy() + private val json: Json by injectLazy() + private val jsonMapper: JsonMapper by injectLazy() + + @Serializable + private data class KoreaderProgressPayload( + val document: String, + val progress: String, + val percentage: Float, + val device: String, + val device_id: String, + ) + + @Serializable + private data class KoreaderProgressResponse( + val document: String? = null, + val progress: String? = null, + val percentage: Float? = null, + val timestamp: Long? = null, + val device: String? = null, + val device_id: String? = null, + ) + + @Serializable + data class SyncResult( + val pageRead: Int, + val timestamp: Long, // Unix timestamp in seconds + val device: String, + val shouldUpdate: Boolean = false, + val isConflict: Boolean = false, + ) + + data class ConnectResult( + val success: Boolean, + val message: String? = null, + val username: String? = null, + ) + + private data class AuthResult( + val success: Boolean, + val message: String? = null, + val isUserNotFoundError: Boolean = false, + ) + + private fun buildRequest( + url: String, + block: Request.Builder.() -> Unit, + ): Request = + Request + .Builder() + .url(url) + .addHeader("Accept", "application/vnd.koreader.v1+json") + .apply(block) + .build() + + private suspend fun getOrGenerateDeviceId(): String { + var deviceId = serverConfig.koreaderSyncDeviceId.value + if (deviceId.isBlank()) { + deviceId = + UUID + .randomUUID() + .toString() + .replace("-", "") + .uppercase() + logger.info { "[KOSYNC] Generated new KOSync Device ID: $deviceId" } + serverConfig.koreaderSyncDeviceId.value = deviceId + } + return deviceId + } + + private fun getOrGenerateChapterHash(chapterId: Int): String? { + return transaction { + val existingHash = + ChapterTable + .select(ChapterTable.koreaderHash) + .where { ChapterTable.id eq chapterId } + .firstOrNull() + ?.get(ChapterTable.koreaderHash) + + if (!existingHash.isNullOrBlank()) { + return@transaction existingHash + } + + val checksumMethod = serverConfig.koreaderSyncChecksumMethod.value + val newHash = + when (checksumMethod) { + KoreaderSyncChecksumMethod.BINARY -> { + logger.info { "[KOSYNC HASH] No hash for chapterId=$chapterId. Generating from CBZ content." } + val mangaId = + ChapterTable + .select(ChapterTable.manga) + .where { ChapterTable.id eq chapterId } + .firstOrNull() + ?.get(ChapterTable.manga) + ?.value ?: return@transaction null + val cbzFile = File(getChapterCbzPath(mangaId, chapterId)) + if (!cbzFile.exists()) { + logger.info { "[KOSYNC HASH] Could not generate hash for chapterId=$chapterId. CBZ not found." } + return@transaction null + } + KoreaderHelper.hashContents(cbzFile) + } + KoreaderSyncChecksumMethod.FILENAME -> { + logger.info { "[KOSYNC HASH] No hash for chapterId=$chapterId. Generating from filename." } + (ChapterTable innerJoin MangaTable) + .select(ChapterTable.name, MangaTable.title) + .where { ChapterTable.id eq chapterId } + .firstOrNull() + ?.let { + val chapterName = it[ChapterTable.name] + val mangaTitle = it[MangaTable.title] + val baseFilename = "$mangaTitle - $chapterName".split('.').dropLast(1).joinToString(".") + Hash.md5(baseFilename) + } + } + } + + if (newHash != null) { + ChapterTable.update({ ChapterTable.id eq chapterId }) { + it[koreaderHash] = newHash + } + logger.info { "[KOSYNC HASH] Generated and saved new hash for chapterId=$chapterId" } + } else { + logger.warn { "[KOSYNC HASH] Hashing failed for chapterId=$chapterId." } + } + newHash + } + } + + private suspend fun register( + username: String, + userkey: String, + ): AuthResult { + val payload = + buildJsonObject { + put("username", username) + put("password", userkey) + } + val request = + buildRequest("${serverConfig.koreaderSyncServerUrl.value.removeSuffix("/")}/users/create") { + post(payload.toString().toRequestBody("application/json".toMediaType())) + } + + return try { + network.client.newCall(request).await().use { response -> + if (response.isSuccessful) { + AuthResult(true, "Registration successful.") + } else { + val errorBody = response.body.string() + val errorMessage = + runCatching { + jsonMapper.fromJsonString>( + errorBody, + Map::class.java, + )["message"] + }.getOrNull() + val finalMessage = errorMessage ?: "Registration failed with code ${response.code}" + AuthResult(false, finalMessage) + } + } + } catch (e: Exception) { + logger.error(e) { "[KOSYNC REGISTER] Exception" } + AuthResult(false, e.message) + } + } + + private suspend fun authorize( + username: String, + userkey: String, + ): AuthResult { + val request = + buildRequest("${serverConfig.koreaderSyncServerUrl.value.removeSuffix("/")}/users/auth") { + get() + addHeader("x-auth-user", username) + addHeader("x-auth-key", userkey) + } + + return try { + network.client.newCall(request).await().use { response -> + if (response.isSuccessful) { + AuthResult(true) + } else { + val isUserNotFound = response.code == 401 // Unauthorized often means user/pass combo is wrong + AuthResult(false, "Authorization failed with code ${response.code}", isUserNotFoundError = isUserNotFound) + } + } + } catch (e: Exception) { + logger.error(e) { "[KOSYNC AUTHORIZE] Exception" } + AuthResult(false, e.message) + } + } + + suspend fun connect( + username: String, + password: String, + ): ConnectResult { + val userkey = Hash.md5(password) + val authResult = authorize(username, userkey) + + if (authResult.success) { + serverConfig.koreaderSyncUsername.value = username + serverConfig.koreaderSyncUserkey.value = userkey + return ConnectResult(true, "Login successful.", username) + } + + if (authResult.isUserNotFoundError) { + logger.info { "[KOSYNC CONNECT] Authorization failed, attempting to register new user." } + val registerResult = register(username, userkey) + return if (registerResult.success) { + serverConfig.koreaderSyncUsername.value = username + serverConfig.koreaderSyncUserkey.value = userkey + ConnectResult(true, "Registration successful.", username) + } else { + ConnectResult(false, registerResult.message ?: "Registration failed.", null) + } + } + + return ConnectResult(false, authResult.message ?: "Authentication failed.", null) + } + + suspend fun logout() { + serverConfig.koreaderSyncUsername.value = "" + serverConfig.koreaderSyncUserkey.value = "" + } + + suspend fun getStatus(): KoSyncStatusPayload { + val username = serverConfig.koreaderSyncUsername.value + val userkey = serverConfig.koreaderSyncUserkey.value + if (username.isBlank() || userkey.isBlank()) { + return KoSyncStatusPayload(isLoggedIn = false, username = null) + } + val authResult = authorize(username, userkey) + return KoSyncStatusPayload(isLoggedIn = authResult.success, username = if (authResult.success) username else null) + } + + suspend fun pushProgress(chapterId: Int) { + val strategy = serverConfig.koreaderSyncStrategy.value + if (strategy == KoreaderSyncStrategy.DISABLED || strategy == KoreaderSyncStrategy.RECEIVE) return + + val username = serverConfig.koreaderSyncUsername.value + val userkey = serverConfig.koreaderSyncUserkey.value + if (username.isBlank() || userkey.isBlank()) return + + logger.info { "[KOSYNC PUSH] Init." } + + val chapterHash = getOrGenerateChapterHash(chapterId) + if (chapterHash.isNullOrBlank()) { + logger.info { "[KOSYNC PUSH] Aborted for chapterId=$chapterId: No hash." } + return + } + + val chapterInfo = + transaction { + ChapterTable + .select(ChapterTable.lastPageRead, ChapterTable.pageCount) + .where { ChapterTable.id eq chapterId } + .firstOrNull() + ?.let { + object { + val lastPageRead = it[ChapterTable.lastPageRead] + val pageCount = it[ChapterTable.pageCount] + } + } + } ?: return + + if (chapterInfo.pageCount <= 0) { + logger.warn { "[KOSYNC PUSH] Aborted for chapterId=$chapterId: Invalid pageCount." } + return + } + + try { + val deviceId = getOrGenerateDeviceId() + val payload = + KoreaderProgressPayload( + document = chapterHash, + progress = (chapterInfo.lastPageRead + 1).toString(), + percentage = (chapterInfo.lastPageRead + 1).toFloat() / chapterInfo.pageCount.toFloat(), + device = "Suwayomi-Server (${System.getProperty("os.name")})", + device_id = deviceId, + ) + + val requestBody = json.encodeToString(KoreaderProgressPayload.serializer(), payload) + val request = + buildRequest("${serverConfig.koreaderSyncServerUrl.value.removeSuffix("/")}/syncs/progress") { + put(requestBody.toRequestBody("application/json".toMediaType())) + addHeader("x-auth-user", username) + addHeader("x-auth-key", userkey) + } + + network.client.newCall(request).await().use { response -> + if (!response.isSuccessful) { + logger.warn { "[KOSYNC PUSH] Failed for chapterId=$chapterId: ${response.code}" } + } else { + logger.info { "[KOSYNC PUSH] Success for chapterId=$chapterId" } + } + } + } catch (e: Exception) { + logger.error(e) { "[KOSYNC PUSH] Exception for chapterId=$chapterId" } + } + } + + suspend fun checkAndPullProgress(chapterId: Int): SyncResult? { + val strategy = serverConfig.koreaderSyncStrategy.value + if (strategy == KoreaderSyncStrategy.DISABLED || strategy == KoreaderSyncStrategy.SEND) return null + + val username = serverConfig.koreaderSyncUsername.value + val userkey = serverConfig.koreaderSyncUserkey.value + if (username.isBlank() || userkey.isBlank()) return null + + val chapterHash = getOrGenerateChapterHash(chapterId) + if (chapterHash.isNullOrBlank()) { + logger.info { "[KOSYNC PULL] Aborted for chapterId=$chapterId: No hash." } + return null + } + + try { + val request = + buildRequest("${serverConfig.koreaderSyncServerUrl.value.removeSuffix("/")}/syncs/progress/$chapterHash") { + get() + addHeader("x-auth-user", username) + addHeader("x-auth-key", userkey) + } + + network.client.newCall(request).await().use { response -> + if (response.isSuccessful) { + val body = response.body.string() + if (body.isBlank() || body == "{}") return null + + val progressResponse = json.decodeFromString(KoreaderProgressResponse.serializer(), body) + val pageRead = progressResponse.progress?.toIntOrNull()?.minus(1) + val timestamp = progressResponse.timestamp + val device = progressResponse.device ?: "KOReader" + + val localProgress = + transaction { + ChapterTable + .select(ChapterTable.lastReadAt, ChapterTable.lastPageRead, ChapterTable.pageCount) + .where { ChapterTable.id eq chapterId } + .firstOrNull() + ?.let { + object { + val lastReadAt = it[ChapterTable.lastReadAt] + val lastPageRead = it[ChapterTable.lastPageRead] + val pageCount = it[ChapterTable.pageCount] + } + } + } + + if (pageRead != null && timestamp != null) { + // Ignore XPath progress for now as we only support paginated files + if (progressResponse.progress?.startsWith("/") == true) { + return null + } + + val localPercentage = + if (localProgress?.pageCount ?: 0 > + 0 + ) { + (localProgress!!.lastPageRead + 1).toFloat() / localProgress.pageCount + } else { + 0f + } + val percentageDifference = abs(localPercentage - (progressResponse.percentage ?: 0f)) + + // Progress is within tolerance, no sync needed + if (percentageDifference < serverConfig.koreaderSyncPercentageTolerance.value) { + return null + } + + when (strategy) { + KoreaderSyncStrategy.RECEIVE -> { + return SyncResult(pageRead, timestamp, device, shouldUpdate = true) + } + KoreaderSyncStrategy.SILENT -> { + if (timestamp > (localProgress?.lastReadAt ?: 0L)) { + return SyncResult(pageRead, timestamp, device, shouldUpdate = true) + } + } + KoreaderSyncStrategy.PROMPT -> { + return SyncResult(pageRead, timestamp, device, isConflict = true) + } + else -> {} // SEND and DISABLED already handled at the start of the function + } + } + } else { + logger.warn { "[KOSYNC PULL] Failed for chapterId=$chapterId: ${response.code}" } + } + } + } catch (e: Exception) { + logger.error(e) { "[KOSYNC PULL] Exception for chapterId=$chapterId" } + } + return null + } +} diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/util/KoreaderHelper.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/util/KoreaderHelper.kt new file mode 100644 index 00000000..2a6ea473 --- /dev/null +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/impl/util/KoreaderHelper.kt @@ -0,0 +1,48 @@ +package suwayomi.tachidesk.manga.impl.util + +import java.io.File +import java.security.MessageDigest + +object KoreaderHelper { + // Helper function to convert ByteArray to Hex String + private fun ByteArray.toHexString() = joinToString("") { "%02x".format(it) } + + /** + * Hashes the document according to a custom Koreader hashing algorithm. + * https://github.com/koreader/koreader/blob/master/frontend/util.lua#L1040 + * Only applies to epub and cbz files. + * @param file The file object to hash. + * @return The lowercase MD5 hash or null if hashing is not possible. + */ + fun hashContents(file: File): String? { + val extension = file.extension.lowercase() + if (!file.exists() || (extension != "epub" && extension != "cbz")) { + return null + } + + try { + file.inputStream().use { fs -> + val step = 1024 + val size = 1024 + val md5 = MessageDigest.getInstance("MD5") + val buffer = ByteArray(size) + + for (i in -1 until 10) { + val position = (step shl (2 * i)).toLong() + if (position >= file.length()) break // Avoid seeking past the end of small files + fs.channel.position(position) + val bytesRead = fs.read(buffer, 0, size) + if (bytesRead > 0) { + md5.update(buffer, 0, bytesRead) + } else { + break + } + } + return md5.digest().toHexString().lowercase() + } + } catch (e: Exception) { + // TODO: Should we log this error? + return null + } + } +} diff --git a/server/src/main/kotlin/suwayomi/tachidesk/manga/model/table/ChapterTable.kt b/server/src/main/kotlin/suwayomi/tachidesk/manga/model/table/ChapterTable.kt index 2e3789df..6b27729a 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/manga/model/table/ChapterTable.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/manga/model/table/ChapterTable.kt @@ -39,6 +39,8 @@ object ChapterTable : IntIdTable() { val pageCount = integer("page_count").default(-1) val manga = reference("manga", MangaTable, ReferenceOption.CASCADE) + + val koreaderHash = varchar("koreader_hash", 32).nullable() } fun ChapterTable.toDataClass( diff --git a/server/src/main/kotlin/suwayomi/tachidesk/opds/impl/OpdsEntryBuilder.kt b/server/src/main/kotlin/suwayomi/tachidesk/opds/impl/OpdsEntryBuilder.kt index b867343c..1f39d575 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/opds/impl/OpdsEntryBuilder.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/opds/impl/OpdsEntryBuilder.kt @@ -6,6 +6,7 @@ import kotlinx.coroutines.withContext import suwayomi.tachidesk.i18n.MR import suwayomi.tachidesk.manga.impl.ChapterDownloadHelper import suwayomi.tachidesk.manga.impl.MangaList.proxyThumbnailUrl +import suwayomi.tachidesk.manga.impl.sync.KoreaderSyncService import suwayomi.tachidesk.manga.model.table.MangaStatus import suwayomi.tachidesk.opds.constants.OpdsConstants import suwayomi.tachidesk.opds.dto.OpdsChapterListAcqEntry @@ -185,7 +186,6 @@ object OpdsEntryBuilder { append(MR.strings.opds_chapter_details_progress.localized(locale, chapter.lastPageRead, chapter.pageCount)) } } - return OpdsEntryXml( id = "urn:suwayomi:chapter:${chapter.id}", title = entryTitle, @@ -222,11 +222,23 @@ object OpdsEntryBuilder { baseUrl: String, locale: Locale, ): OpdsEntryXml { + // Check remote progress before building the entry + val syncResult = KoreaderSyncService.checkAndPullProgress(chapter.id) + + val localLastPageRead = chapter.lastPageRead + val remoteLastPageRead = syncResult?.pageRead + val remoteLastReadAt = syncResult?.timestamp + + val showConflict = syncResult?.isConflict == true && remoteLastPageRead != null && localLastPageRead != remoteLastPageRead + + val finalLastPageRead = if (syncResult?.shouldUpdate == true) remoteLastPageRead ?: localLastPageRead else localLastPageRead + val finalLastReadAt = if (syncResult?.shouldUpdate == true) remoteLastReadAt ?: chapter.lastReadAt else chapter.lastReadAt + val statusKey = when { chapter.downloaded -> MR.strings.opds_chapter_status_downloaded chapter.read -> MR.strings.opds_chapter_status_read - chapter.lastPageRead > 0 -> MR.strings.opds_chapter_status_in_progress + finalLastPageRead > 0 -> MR.strings.opds_chapter_status_in_progress else -> MR.strings.opds_chapter_status_unread } val titlePrefix = statusKey.localized(locale) @@ -238,7 +250,7 @@ object OpdsEntryBuilder { append(MR.strings.opds_chapter_details_scanlator.localized(locale, it)) } val pageCountDisplay = chapter.pageCount.takeIf { it > 0 } ?: "?" - append(MR.strings.opds_chapter_details_progress.localized(locale, chapter.lastPageRead, pageCountDisplay)) + append(MR.strings.opds_chapter_details_progress.localized(locale, finalLastPageRead, pageCountDisplay)) } val links = mutableListOf() var cbzFileSize: Long? = null @@ -270,19 +282,73 @@ object OpdsEntryBuilder { } } if (chapter.pageCount > 0) { - links.add( - OpdsLinkXml( - rel = OpdsConstants.LINK_REL_PSE_STREAM, - href = - "/api/v1/manga/${manga.id}/chapter/${chapter.sourceOrder}/page/{pageNumber}" + - "?updateProgress=${serverConfig.opdsEnablePageReadProgress.value}", - type = OpdsConstants.TYPE_IMAGE_JPEG, - title = MR.strings.opds_linktitle_stream_pages.localized(locale), - pseCount = chapter.pageCount, - pseLastRead = chapter.lastPageRead.takeIf { it > 0 }, - pseLastReadDate = chapter.lastReadAt.takeIf { it > 0 }?.let { OpdsDateUtil.formatEpochMillisForOpds(it * 1000) }, - ), - ) + val basePageHref = + "/api/v1/manga/${manga.id}/chapter/${chapter.sourceOrder}/page/{pageNumber}" + + "?updateProgress=${serverConfig.opdsEnablePageReadProgress.value}" + + if (showConflict) { + // Option 1: Local progress + val localTitleRes = + if (localLastPageRead > + 0 + ) { + MR.strings.opds_linktitle_stream_pages_continue_local + } else { + MR.strings.opds_linktitle_stream_pages_start_local + } + links.add( + OpdsLinkXml( + rel = OpdsConstants.LINK_REL_PSE_STREAM, + href = basePageHref, + type = OpdsConstants.TYPE_IMAGE_JPEG, + title = localTitleRes.localized(locale), + pseCount = chapter.pageCount, + pseLastRead = localLastPageRead.takeIf { it >= 0 }, + pseLastReadDate = chapter.lastReadAt.takeIf { it > 0 }?.let { OpdsDateUtil.formatEpochMillisForOpds(it * 1000) }, + ), + ) + // Option 2: Remote progress + val remoteTitleRes = + if (remoteLastPageRead > + 0 + ) { + MR.strings.opds_linktitle_stream_pages_continue_remote + } else { + MR.strings.opds_linktitle_stream_pages_start_remote + } + links.add( + OpdsLinkXml( + rel = OpdsConstants.LINK_REL_PSE_STREAM, + href = basePageHref, + type = OpdsConstants.TYPE_IMAGE_JPEG, + title = remoteTitleRes.localized(locale, syncResult.device), + pseCount = chapter.pageCount, + pseLastRead = remoteLastPageRead, + pseLastReadDate = remoteLastReadAt?.takeIf { it > 0 }?.let { OpdsDateUtil.formatEpochMillisForOpds(it * 1000) }, + ), + ) + } else { + // Normal behavior: single progress link + val titleRes = + if (finalLastPageRead > + 0 + ) { + MR.strings.opds_linktitle_stream_pages_continue + } else { + MR.strings.opds_linktitle_stream_pages_start + } + links.add( + OpdsLinkXml( + rel = OpdsConstants.LINK_REL_PSE_STREAM, + href = basePageHref, + type = OpdsConstants.TYPE_IMAGE_JPEG, + title = titleRes.localized(locale), + pseCount = chapter.pageCount, + pseLastRead = finalLastPageRead.takeIf { it > 0 }, + pseLastReadDate = finalLastReadAt.takeIf { it > 0 }?.let { OpdsDateUtil.formatEpochMillisForOpds(it * 1000) }, + ), + ) + } links.add( OpdsLinkXml( rel = OpdsConstants.LINK_REL_IMAGE, diff --git a/server/src/main/kotlin/suwayomi/tachidesk/server/ServerConfig.kt b/server/src/main/kotlin/suwayomi/tachidesk/server/ServerConfig.kt index 68a31935..36a433df 100644 --- a/server/src/main/kotlin/suwayomi/tachidesk/server/ServerConfig.kt +++ b/server/src/main/kotlin/suwayomi/tachidesk/server/ServerConfig.kt @@ -26,6 +26,8 @@ import kotlinx.coroutines.flow.mapLatest import kotlinx.coroutines.flow.onEach import org.jetbrains.exposed.sql.SortOrder import suwayomi.tachidesk.graphql.types.AuthMode +import suwayomi.tachidesk.graphql.types.KoreaderSyncChecksumMethod +import suwayomi.tachidesk.graphql.types.KoreaderSyncStrategy import suwayomi.tachidesk.graphql.types.WebUIChannel import suwayomi.tachidesk.graphql.types.WebUIFlavor import suwayomi.tachidesk.graphql.types.WebUIInterface @@ -195,6 +197,15 @@ class ServerConfig( val opdsShowOnlyDownloadedChapters: MutableStateFlow by OverrideConfigValue() val opdsChapterSortOrder: MutableStateFlow by OverrideConfigValue() + // koreader sync + val koreaderSyncServerUrl: MutableStateFlow by OverrideConfigValue() + val koreaderSyncUsername: MutableStateFlow by OverrideConfigValue() + val koreaderSyncUserkey: MutableStateFlow by OverrideConfigValue() + val koreaderSyncDeviceId: MutableStateFlow by OverrideConfigValue() + val koreaderSyncChecksumMethod: MutableStateFlow by OverrideConfigValue() + val koreaderSyncStrategy: MutableStateFlow by OverrideConfigValue() + val koreaderSyncPercentageTolerance: MutableStateFlow by OverrideConfigValue() + @OptIn(ExperimentalCoroutinesApi::class) fun subscribeTo( flow: Flow, diff --git a/server/src/main/kotlin/suwayomi/tachidesk/server/database/migration/M0051_AddKoreaderHashToChapterTable.kt b/server/src/main/kotlin/suwayomi/tachidesk/server/database/migration/M0051_AddKoreaderHashToChapterTable.kt new file mode 100644 index 00000000..1abff057 --- /dev/null +++ b/server/src/main/kotlin/suwayomi/tachidesk/server/database/migration/M0051_AddKoreaderHashToChapterTable.kt @@ -0,0 +1,19 @@ +package suwayomi.tachidesk.server.database.migration + +/* + * Copyright (C) Contributors to the Suwayomi project + * + * This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at https://mozilla.org/MPL/2.0/. */ + +import de.neonew.exposed.migrations.helpers.AddColumnMigration + +@Suppress("ClassName", "unused") +class M0051_AddKoreaderHashToChapterTable : + AddColumnMigration( + "Chapter", + "koreader_hash", + "VARCHAR(32)", + "NULL", + ) diff --git a/server/src/main/resources/server-reference.conf b/server/src/main/resources/server-reference.conf index 6242b450..ec1018b5 100644 --- a/server/src/main/resources/server-reference.conf +++ b/server/src/main/resources/server-reference.conf @@ -85,3 +85,12 @@ server.opdsMarkAsReadOnDownload = false server.opdsShowOnlyUnreadChapters = false server.opdsShowOnlyDownloadedChapters = false server.opdsChapterSortOrder = "DESC" # "ASC", "DESC" + +# Koreader Sync +server.koreaderSyncServerUrl = "http://localhost:17200" +server.koreaderSyncUsername = "" +server.koreaderSyncUserkey = "" +server.koreaderSyncDeviceId = "" +server.koreaderSyncChecksumMethod = "binary" # "binary" or "filename" +server.koreaderSyncStrategy = "disabled" # "prompt", "silent", "send", "receive", "disabled" +server.koreaderSyncPercentageTolerance = 0.00000000000001 # absolute tolerance for progress comparison from 1 (widest) to 1e-15 (strict)