Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit e5360fbf authored by Jeff Brown's avatar Jeff Brown
Browse files

Rewrite SQLite database wrappers.

The main theme of this change is encapsulation.  This change
preserves all existing functionality but the implementation
is now much cleaner.

Instead of a "database lock", access to the database is treated
as a resource acquisition problem.  If a thread's owns a database
connection, then it can access the database; otherwise, it must
acquire a database connection first, and potentially wait for other
threads to give up theirs.  The SQLiteConnectionPool encapsulates
the details of how connections are created, configured, acquired,
released and disposed.

One new feature is that SQLiteConnectionPool can make scheduling
decisions about which thread should next acquire a database
connection when there is contention among threads.  The factors
considered include wait queue ordering (fairness among peers),
whether the connection is needed for an interactive operation
(unfairness on behalf of the UI), and whether the primary connection
is needed or if any old connection will do.  Thus one goal of the
new SQLiteConnectionPool is to improve the utilization of
database connections.

To emulate some quirks of the old "database lock," we introduce
the concept of the primary database connection.  The primary
database connection is the one that is typically used to perform
write operations to the database.  When a thread holds the primary
database connection, it effectively prevents other threads from
modifying the database (although they can still read).  What's
more, those threads will block when they try to acquire the primary
connection, which provides the same kind of mutual exclusion
features that the old "database lock" had.  (In truth, we
probably don't need to be requiring use of the primary database
connection in as many places as we do now, but we can seek to refine
that behavior in future patches.)

Another significant change is that native sqlite3_stmt objects
(prepared statements) are fully encapsulated by the SQLiteConnection
object that owns them.  This ensures that the connection can
finalize (destroy) all extant statements that belong to a database
connection when the connection is closed.  (In the original code,
this was very complicated because the sqlite3_stmt objects were
managed by SQLiteCompiledSql objects which had different lifetime
from the original SQLiteDatabase that created them.  Worse, the
SQLiteCompiledSql finalizer method couldn't actually destroy the
sqlite3_stmt objects because it ran on the finalizer thread and
therefore could not guarantee that it could acquire the database
lock in order to do the work.  This resulted in some rather
tortured logic involving a list of pending finalizable statements
and a high change of deadlocks or leaks.)

Because sqlite3_stmt objects never escape the confines of the
SQLiteConnection that owns them, we can also greatly simplify
the design of the SQLiteProgram, SQLiteQuery and SQLiteStatement
objects.  They no longer have to wrangle a native sqlite3_stmt
object pointer and manage its lifecycle.  So now all they do
is hold bind arguments and provide a fancy API.

All of the JNI glue related to managing database connections
and performing transactions is now bound to SQLiteConnection
(rather than being scattered everywhere).  This makes sense because
SQLiteConnection owns the native sqlite3 object, so it is the
only class in the system that can interact with the native
SQLite database directly.  Encapsulation for the win.

One particularly tricky part of this change is managing the
ownership of SQLiteConnection objects.  At any given time,
a SQLiteConnection is either owned by a SQLiteConnectionPool
or by a SQLiteSession.  SQLiteConnections should never be leaked,
but we handle that case too (and yell about it with CloseGuard).

A SQLiteSession object is responsible for acquiring and releasing
a SQLiteConnection object on behalf of a single thread as needed.
For example, the session acquires a connection when a transaction
begins and releases it when finished.  If the session cannot
acquire a connection immediately, then the requested operation
blocks until a connection becomes available.

SQLiteSessions are thread-local.  A SQLiteDatabase assigns a
distinct session to each thread that performs database operations.
This is very very important.  First, it prevents two threads
from trying to use the same SQLiteConnection at the same time
(because two threads can't share the same session).
Second, it prevents a single thread from trying to acquire two
SQLiteConnections simultaneously from the same database (because
a single thread can't have two sessions for the same database which,
in addition to being greedy, could result in a deadlock).

There is strict layering between the various database objects,
objects at lower layers are not aware of objects at higher layers.
Moreover, objects at higher layers generally own objects at lower
layers and are responsible for ensuring they are properly disposed
when no longer needed (good for the environment).

API layer: SQLiteDatabase, SQLiteProgram, SQLiteQuery, SQLiteStatement.
Session layer: SQLiteSession.
Connection layer: SQLiteConnectionPool, SQLiteConnection.
Native layer: JNI glue.

By avoiding cyclic dependencies between layers, we make the
architecture much more intelligible, maintainable and robust.

Finally, this change adds a great deal of new debugging information.
It is now possible to view a list of the most recent database
operations including how long they took to run using
"adb shell dumpsys dbinfo".  (Because most of the interesting
work happens in SQLiteConnection, it is easy to add debugging
instrumentation to track all database operations in one place.)

Change-Id: Iffb4ce72d8bcf20b4e087d911da6aa84d2f15297
parent 3f11f302
Loading
Loading
Loading
Loading
+2 −2
Original line number Diff line number Diff line
@@ -7204,7 +7204,7 @@ package android.database.sqlite {
    method public long insertWithOnConflict(java.lang.String, java.lang.String, android.content.ContentValues, int);
    method public boolean isDatabaseIntegrityOk();
    method public boolean isDbLockedByCurrentThread();
    method public boolean isDbLockedByOtherThreads();
    method public deprecated boolean isDbLockedByOtherThreads();
    method public boolean isOpen();
    method public boolean isReadOnly();
    method public deprecated void markTableSyncable(java.lang.String, java.lang.String);
@@ -7226,7 +7226,7 @@ package android.database.sqlite {
    method public long replace(java.lang.String, java.lang.String, android.content.ContentValues);
    method public long replaceOrThrow(java.lang.String, java.lang.String, android.content.ContentValues) throws android.database.SQLException;
    method public void setLocale(java.util.Locale);
    method public void setLockingEnabled(boolean);
    method public deprecated void setLockingEnabled(boolean);
    method public void setMaxSqlCacheSize(int);
    method public long setMaximumSize(long);
    method public void setPageSize(long);
+0 −348
Original line number Diff line number Diff line
/*
 * Copyright (C) 20010 The Android Open Source Project
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package android.database.sqlite;

import android.content.res.Resources;
import android.os.SystemClock;
import android.util.Log;

import java.util.ArrayList;
import java.util.HashSet;
import java.util.Random;

/**
 * A connection pool to be used by readers.
 * Note that each connection can be used by only one reader at a time.
 */
/* package */ class DatabaseConnectionPool {

    private static final String TAG = "DatabaseConnectionPool";

    /** The default connection pool size. */
    private volatile int mMaxPoolSize =
        Resources.getSystem().getInteger(com.android.internal.R.integer.db_connection_pool_size);

    /** The connection pool objects are stored in this member.
     * TODO: revisit this data struct as the number of pooled connections increase beyond
     * single-digit values.
     */
    private final ArrayList<PoolObj> mPool = new ArrayList<PoolObj>(mMaxPoolSize);

    /** the main database connection to which this connection pool is attached */
    private final SQLiteDatabase mParentDbObj;

    /** Random number generator used to pick a free connection out of the pool */
    private Random rand; // lazily initialized

    /* package */ DatabaseConnectionPool(SQLiteDatabase db) {
        this.mParentDbObj = db;
        if (Log.isLoggable(TAG, Log.DEBUG)) {
            Log.d(TAG, "Max Pool Size: " + mMaxPoolSize);
        }
    }

    /**
     * close all database connections in the pool - even if they are in use!
     */
    /* package */ synchronized void close() {
        if (Log.isLoggable(TAG, Log.DEBUG)) {
            Log.d(TAG, "Closing the connection pool on " + mParentDbObj.getPath() + toString());
        }
        for (int i = mPool.size() - 1; i >= 0; i--) {
            mPool.get(i).mDb.close();
        }
        mPool.clear();
    }

    /**
     * get a free connection from the pool
     *
     * @param sql if not null, try to find a connection inthe pool which already has cached
     * the compiled statement for this sql.
     * @return the Database connection that the caller can use
     */
    /* package */ synchronized SQLiteDatabase get(String sql) {
        SQLiteDatabase db = null;
        PoolObj poolObj = null;
        int poolSize = mPool.size();
        if (Log.isLoggable(TAG, Log.DEBUG)) {
            assert sql != null;
            doAsserts();
        }
        if (getFreePoolSize() == 0) {
            // no free ( = available) connections
            if (mMaxPoolSize == poolSize) {
                // maxed out. can't open any more connections.
                // let the caller wait on one of the pooled connections
                // preferably a connection caching the pre-compiled statement of the given SQL
                if (mMaxPoolSize == 1) {
                    poolObj = mPool.get(0);
                } else {
                    for (int i = 0; i < mMaxPoolSize; i++) {
                        if (mPool.get(i).mDb.isInStatementCache(sql)) {
                            poolObj = mPool.get(i);
                            break;
                        }
                    }
                    if (poolObj == null) {
                        // there are no database connections with the given SQL pre-compiled.
                        // ok to return any of the connections.
                        if (rand == null) {
                            rand = new Random(SystemClock.elapsedRealtime());
                        }
                        poolObj = mPool.get(rand.nextInt(mMaxPoolSize));
                    }
                }
                db = poolObj.mDb;
            } else {
                // create a new connection and add it to the pool, since we haven't reached
                // max pool size allowed
                db = mParentDbObj.createPoolConnection((short)(poolSize + 1));
                poolObj = new PoolObj(db);
                mPool.add(poolSize, poolObj);
            }
        } else {
            // there are free connections available. pick one
            // preferably a connection caching the pre-compiled statement of the given SQL
            for (int i = 0; i < poolSize; i++) {
                if (mPool.get(i).isFree() && mPool.get(i).mDb.isInStatementCache(sql)) {
                    poolObj = mPool.get(i);
                    break;
                }
            }
            if (poolObj == null) {
                // didn't find a free database connection with the given SQL already
                // pre-compiled. return a free connection (this means, the same SQL could be
                // pre-compiled on more than one database connection. potential wasted memory.)
                for (int i = 0; i < poolSize; i++) {
                    if (mPool.get(i).isFree()) {
                        poolObj = mPool.get(i);
                        break;
                    }
                }
            }
            db = poolObj.mDb;
        }

        assert poolObj != null;
        assert poolObj.mDb == db;

        poolObj.acquire();
        if (Log.isLoggable(TAG, Log.DEBUG)) {
            Log.d(TAG, "END get-connection: " + toString() + poolObj.toString());
        }
        return db;
        // TODO if a thread acquires a connection and dies without releasing the connection, then
        // there could be a connection leak.
    }

    /**
     * release the given database connection back to the pool.
     * @param db the connection to be released
     */
    /* package */ synchronized void release(SQLiteDatabase db) {
        if (Log.isLoggable(TAG, Log.DEBUG)) {
            assert db.mConnectionNum > 0;
            doAsserts();
            assert mPool.get(db.mConnectionNum - 1).mDb == db;
        }

        PoolObj poolObj = mPool.get(db.mConnectionNum - 1);

        if (Log.isLoggable(TAG, Log.DEBUG)) {
            Log.d(TAG, "BEGIN release-conn: " + toString() + poolObj.toString());
        }

        if (poolObj.isFree()) {
            throw new IllegalStateException("Releasing object already freed: " +
                    db.mConnectionNum);
        }

        poolObj.release();
        if (Log.isLoggable(TAG, Log.DEBUG)) {
            Log.d(TAG, "END release-conn: " + toString() + poolObj.toString());
        }
    }

    /**
     * Returns a list of all database connections in the pool (both free and busy connections).
     * This method is used when "adb bugreport" is done.
     */
    /* package */ synchronized ArrayList<SQLiteDatabase> getConnectionList() {
        ArrayList<SQLiteDatabase> list = new ArrayList<SQLiteDatabase>();
        for (int i = mPool.size() - 1; i >= 0; i--) {
            list.add(mPool.get(i).mDb);
        }
        return list;
    }

    /**
     * package level access for testing purposes only. otherwise, private should be sufficient.
     */
    /* package */ int getFreePoolSize() {
        int count = 0;
        for (int i = mPool.size() - 1; i >= 0; i--) {
            if (mPool.get(i).isFree()) {
                count++;
            }
        }
        return count++;
    }

    /**
     * only for testing purposes
     */
    /* package */ ArrayList<PoolObj> getPool() {
        return mPool;
    }

    @Override
    public String toString() {
        StringBuilder buff = new StringBuilder();
        buff.append("db: ");
        buff.append(mParentDbObj.getPath());
        buff.append(", totalsize = ");
        buff.append(mPool.size());
        buff.append(", #free = ");
        buff.append(getFreePoolSize());
        buff.append(", maxpoolsize = ");
        buff.append(mMaxPoolSize);
        for (PoolObj p : mPool) {
            buff.append("\n");
            buff.append(p.toString());
        }
        return buff.toString();
    }

    private void doAsserts() {
        for (int i = 0; i < mPool.size(); i++) {
            mPool.get(i).verify();
            assert mPool.get(i).mDb.mConnectionNum == (i + 1);
        }
    }

    /** only used for testing purposes. */
    /* package */ synchronized void setMaxPoolSize(int size) {
        mMaxPoolSize = size;
    }

    /** only used for testing purposes. */
    /* package */ synchronized int getMaxPoolSize() {
        return mMaxPoolSize;
    }

    /** only used for testing purposes. */
    /* package */ boolean isDatabaseObjFree(SQLiteDatabase db) {
        return mPool.get(db.mConnectionNum - 1).isFree();
    }

    /** only used for testing purposes. */
    /* package */ int getSize() {
        return mPool.size();
    }

    /**
     * represents objects in the connection pool.
     * package-level access for testing purposes only.
     */
    /* package */ static class PoolObj {

        private final SQLiteDatabase mDb;
        private boolean mFreeBusyFlag = FREE;
        private static final boolean FREE = true;
        private static final boolean BUSY = false;

        /** the number of threads holding this connection */
        // @GuardedBy("this")
        private int mNumHolders = 0;

        /** contains the threadIds of the threads holding this connection.
         * used for debugging purposes only.
         */
        // @GuardedBy("this")
        private HashSet<Long> mHolderIds = new HashSet<Long>();

        public PoolObj(SQLiteDatabase db) {
            mDb = db;
        }

        private synchronized void acquire() {
            if (Log.isLoggable(TAG, Log.DEBUG)) {
                assert isFree();
                long id = Thread.currentThread().getId();
                assert !mHolderIds.contains(id);
                mHolderIds.add(id);
            }

            mNumHolders++;
            mFreeBusyFlag = BUSY;
        }

        private synchronized void release() {
            if (Log.isLoggable(TAG, Log.DEBUG)) {
                long id = Thread.currentThread().getId();
                assert mHolderIds.size() == mNumHolders;
                assert mHolderIds.contains(id);
                mHolderIds.remove(id);
            }

            mNumHolders--;
            if (mNumHolders == 0) {
                mFreeBusyFlag = FREE;
            }
        }

        private synchronized boolean isFree() {
            if (Log.isLoggable(TAG, Log.DEBUG)) {
                verify();
            }
            return (mFreeBusyFlag == FREE);
        }

        private synchronized void verify() {
            if (mFreeBusyFlag == FREE) {
                assert mNumHolders == 0;
            } else {
                assert mNumHolders > 0;
            }
        }

        /**
         * only for testing purposes
         */
        /* package */ synchronized int getNumHolders() {
            return mNumHolders;
        }

        @Override
        public String toString() {
            StringBuilder buff = new StringBuilder();
            buff.append(", conn # ");
            buff.append(mDb.mConnectionNum);
            buff.append(", mCountHolders = ");
            synchronized(this) {
                buff.append(mNumHolders);
                buff.append(", freeBusyFlag = ");
                buff.append(mFreeBusyFlag);
                for (Long l : mHolderIds) {
                    buff.append(", id = " + l);
                }
            }
            return buff.toString();
        }
    }
}
+1 −21
Original line number Diff line number Diff line
@@ -16,8 +16,6 @@

package android.database.sqlite;

import android.database.CursorWindow;

/**
 * An object created from a SQLiteDatabase that can be closed.
 */
@@ -31,7 +29,7 @@ public abstract class SQLiteClosable {
        synchronized(this) {
            if (mReferenceCount <= 0) {
                throw new IllegalStateException(
                        "attempt to re-open an already-closed object: " + getObjInfo());
                        "attempt to re-open an already-closed object: " + this);
            }
            mReferenceCount++;
        }
@@ -56,22 +54,4 @@ public abstract class SQLiteClosable {
            onAllReferencesReleasedFromContainer();
        }
    }

    private String getObjInfo() {
        StringBuilder buff = new StringBuilder();
        buff.append(this.getClass().getName());
        buff.append(" (");
        if (this instanceof SQLiteDatabase) {
            buff.append("database = ");
            buff.append(((SQLiteDatabase)this).getPath());
        } else if (this instanceof SQLiteProgram) {
            buff.append("mSql = ");
            buff.append(((SQLiteProgram)this).mSql);
        } else if (this instanceof CursorWindow) {
            buff.append("mStartPos = ");
            buff.append(((CursorWindow)this).getStartPosition());
        }
        buff.append(") ");
        return buff.toString();
    }
}
+0 −158
Original line number Diff line number Diff line
/*
 * Copyright (C) 2009 The Android Open Source Project
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package android.database.sqlite;

import android.os.StrictMode;
import android.util.Log;

/**
 * This class encapsulates compilation of sql statement and release of the compiled statement obj.
 * Once a sql statement is compiled, it is cached in {@link SQLiteDatabase}
 * and it is released in one of the 2 following ways
 * 1. when {@link SQLiteDatabase} object is closed.
 * 2. if this is not cached in {@link SQLiteDatabase}, {@link android.database.Cursor#close()}
 * releaases this obj.
 */
/* package */ class SQLiteCompiledSql {

    private static final String TAG = "SQLiteCompiledSql";

    /** The database this program is compiled against. */
    /* package */ final SQLiteDatabase mDatabase;

    /**
     * Native linkage, do not modify. This comes from the database.
     */
    /* package */ final int nHandle;

    /**
     * Native linkage, do not modify. When non-0 this holds a reference to a valid
     * sqlite3_statement object. It is only updated by the native code, but may be
     * checked in this class when the database lock is held to determine if there
     * is a valid native-side program or not.
     */
    /* package */ int nStatement = 0;

    /** the following are for debugging purposes */
    private String mSqlStmt = null;
    private final Throwable mStackTrace;

    /** when in cache and is in use, this member is set */
    private boolean mInUse = false;

    /* package */ SQLiteCompiledSql(SQLiteDatabase db, String sql) {
        db.verifyDbIsOpen();
        db.verifyLockOwner();
        mDatabase = db;
        mSqlStmt = sql;
        if (StrictMode.vmSqliteObjectLeaksEnabled()) {
            mStackTrace = new DatabaseObjectNotClosedException().fillInStackTrace();
        } else {
            mStackTrace = null;
        }
        nHandle = db.mNativeHandle;
        native_compile(sql);
    }

    /* package */ void releaseSqlStatement() {
        // Note that native_finalize() checks to make sure that nStatement is
        // non-null before destroying it.
        if (nStatement != 0) {
            mDatabase.finalizeStatementLater(nStatement);
            nStatement = 0;
        }
    }

    /**
     * returns true if acquire() succeeds. false otherwise.
     */
    /* package */ synchronized boolean acquire() {
        if (mInUse) {
            // it is already in use.
            return false;
        }
        mInUse = true;
        return true;
    }

    /* package */ synchronized void release() {
        mInUse = false;
    }

    /* package */ synchronized void releaseIfNotInUse() {
        // if it is not in use, release its memory from the database
        if (!mInUse) {
            releaseSqlStatement();
        }
    }

    /**
     * Make sure that the native resource is cleaned up.
     */
    @Override
    protected void finalize() throws Throwable {
        try {
            if (nStatement == 0) return;
            // don't worry about finalizing this object if it is ALREADY in the
            // queue of statements to be finalized later
            if (mDatabase.isInQueueOfStatementsToBeFinalized(nStatement)) {
                return;
            }
            // finalizer should NEVER get called
            // but if the database itself is not closed and is GC'ed, then
            // all sub-objects attached to the database could end up getting GC'ed too.
            // in that case, don't print any warning.
            if (mInUse && mStackTrace != null) {
                int len = mSqlStmt.length();
                StrictMode.onSqliteObjectLeaked(
                    "Releasing statement in a finalizer. Please ensure " +
                    "that you explicitly call close() on your cursor: " +
                    mSqlStmt.substring(0, (len > 1000) ? 1000 : len),
                    mStackTrace);
            }
            releaseSqlStatement();
        } finally {
            super.finalize();
        }
    }

    @Override public String toString() {
        synchronized(this) {
            StringBuilder buff = new StringBuilder();
            buff.append(" nStatement=");
            buff.append(nStatement);
            buff.append(", mInUse=");
            buff.append(mInUse);
            buff.append(", db=");
            buff.append(mDatabase.getPath());
            buff.append(", db_connectionNum=");
            buff.append(mDatabase.mConnectionNum);
            buff.append(", sql=");
            int len = mSqlStmt.length();
            buff.append(mSqlStmt.substring(0, (len > 100) ? 100 : len));
            return buff.toString();
        }
    }

    /**
     * Compiles SQL into a SQLite program.
     *
     * <P>The database lock must be held when calling this method.
     * @param sql The SQL to compile.
     */
    private final native void native_compile(String sql);
}
+1149 −0

File added.

Preview size limit exceeded, changes collapsed.

Loading