Avatar billede mortennp Nybegynder
08. januar 2009 - 22:02 Der er 6 kommentarer

Problem med at eksekvere query pga. "hukommelse"?

Jeg er ved at forsøge at konvertere fra SQL server til MySQL. Foreløbigt har jeg forsøgt at loade en CSV fil ind på 500 mb i MySQL. Det har jeg ingen problemer i, men bruger jeg derefter en "select * from.." kommando på selvsamme tabel, får jeg følgende meddelelse efter et stykke tid:

"The memory load of the system is extremely high. This is likely because of the current result being very large. In order to keep the system responsive set retrieval has been stopped."

Dette giver yderligere fejlnr. 1317 "Query execution interrupted."

Jeg har installeret MySQL og ikke pillet ved indstillingerne. Jeg forstår dog stadig ikke at jeg skulle få fejlen, på samme pc har jeg med SQL Server loadet samme fil ind samt en anden fil der sammen med den jeg forsøger mig med nu udgør den samlede tabel.SQL serveren har ingen problemer med at kører min query forespørgsel, så jeg går ud fra det også må være muligt med MySQL, men hvad skal jeg gøre for ikke at få hukommelsesfejlen?

Håber rigtigt meget på I kan hjælpe mig, da jeg ikke umiddelbart kan se hvad jeg gør forkert :-)

På forhånd tak

Mvh

Morten
Avatar billede arne_v Ekspert
08. januar 2009 - 22:18 #1
Nu er det ko nok ikke fysisk memory som er problemet.

Snarere det du har fortalt MySQL at den må bruge.

Check mysql.ini, mysqld sektion, diverse buffer størrelser.
Avatar billede mortennp Nybegynder
08. januar 2009 - 22:55 #2
Hej Arne,

Tak for dit svar.

Jeg må indrømme at jeg endnu ikke er supermeget inde i MySQL, men jeg synes det lyder logisk som du siger, at det nok ikke er den fysiske hukommelse, for så burde SQL server vel heller ikke kunne klare en query der er dobbelt så stor.

Mht. mysql.ini så har jeg fundet den og kigget i den, men det må jeg indrømme at jeg ikke bliver meget klogere af. Der er et uddrag af det der umiddelbart er relevant her:

----------------------------------

# SERVER SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this
# file.
#
[mysqld]

# The TCP/IP Port the MySQL Server will listen on
port=3306


#Path to installation directory. All paths are usually resolved relative to this.
basedir="C:/Programmer/MySQL/MySQL Server 5.1/"

#Path to the database root
datadir="C:/Documents and Settings/All Users/Application Data/MySQL/MySQL Server 5.1/Data/"

# The default character set that will be used when a new schema or table is
# created and no character set is defined
default-character-set=latin1

# The default storage engine that will be used when create new tables when
default-storage-engine=INNODB

# Set the SQL mode to strict
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections=100

# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
query_cache_size=0

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_cache=256

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=93M


# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=8


#*** INNODB Specific options ***


# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
#skip-innodb

# Additional memory pool that is used by InnoDB to store metadata
# information.  If InnoDB requires more memory for this purpose it will
# start to allocate it from the OS.  As this is fast enough on most
# recent operating systems, you normally do not need to change this
# value. SHOW INNODB STATUS will display the current amount used.
innodb_additional_mem_pool_size=7M

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit=1

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=4M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=304M

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
innodb_log_file_size=10M

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=8

-------------

Hjælper det noget?

Hvor finder jeg mysqld sektionen henne?

Jeg har prøvet at kigge i MySQL administrator og justeret på

-"Buffer pool size" under innoDB Parameters og sat den til 999 mb

-"Key buffer" under general parameters til 600mb.

Men det er ikke umiddelbart gået godt?
Avatar billede arne_v Ekspert
08. januar 2009 - 23:00 #3
[mysqld] start på mysqld sektionen og den fortsætter til næste [xxx]

Er tabellerne InnoDB tabeller ?
Avatar billede arne_v Ekspert
08. januar 2009 - 23:02 #4
Umiddelbart vulle jeg gætte på enten query_cache_size eller innodb_buffer_pool_size (det sidste
forudsætter at det er en InnoDB tabel).
Avatar billede mortennp Nybegynder
08. januar 2009 - 23:37 #5
Ja tabellen er en InnoDB,

Ved innodb_buffer_pool_size 100 mb returneres 324917 rækker

Ved 600mb returneres 282832 rækker

Ved 700mb returneres 289321 rækker

Så umiddelbart virker det ikke til at være innodb buffer size'n der gør om queryen kan køres eller ej?

Ovenstående er forudsat at det er korrekt "bare" at redigere i my.ini og derefter genstarte mysql servicen.
Avatar billede arne_v Ekspert
08. januar 2009 - 23:59 #6
Måske har du ramt en query hvor MySQL bare er mindre effektiv end MySQL.

Har du eksperimenetert med query_cache_size ?
Avatar billede Ny bruger Nybegynder

Din løsning...

Tilladte BB-code-tags: [b]fed[/b] [i]kursiv[/i] [u]understreget[/u] Web- og emailadresser omdannes automatisk til links. Der sættes "nofollow" på alle links.

Loading billede Opret Preview
Kategori
Computerworld tilbyder specialiserede kurser i database-management

Log ind eller opret profil

Hov!

For at kunne deltage på Computerworld Eksperten skal du være logget ind.

Det er heldigvis nemt at oprette en bruger: Det tager to minutter og du kan vælge at bruge enten e-mail, Facebook eller Google som login.

Du kan også logge ind via nedenstående tjenester